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1.  INTRODUCTI  ON:  Narrative  that  briefly  (one  paragraph)  describes  the  subject,  purpose  and 
scope  of  the  research. 

This  project  will  determine  how  frequency  of  exposure  to  targeted  clinical  problems  contributes 
uniquely,  and  combined  with  other  identified  variables,  to  affect  decay  in  physicians’  cognitive  clinical 
skills.  We  have  targeted  9  clinical  problems  that  fulfill  specific  a-priori  criteria  and  have  identified 
critical  elements  of  the  care  process  that  demonstrate  evidence  of  skill  decay  with  increasing  time  since 
most  recent  exposure  to  that  clinical  problem.  We  continue  to  evaluate  new  data  sources  that  may 
enhance  the  specification  of  explanatory  models.  We  are  refining  multivariate  quantitative  models  to 
determine  if  skill  decay  can  be  reliably  predicted  based  on  specification  of  these  other  variables, 
including  frequency  of  exposure.  In  Year  3,  we  will  test  behavioral,  psychoeducational  or  electronic 
interventions  to  minimize  decay  in  physicians’  cognitive  clinical  skills.  The  project  is  well-positioned 
to  be  completed  effectively,  productively  and  within  budget. 

2.  KEYWORDS:  Provide  a  brief  list  of  keywords  (limit  to  20  words). 


Health  care  quality  and  safety;  skill  decay;  physicians;  declarative  knowledge;  clinical  decision 
making;  decision  science 


3.  ACCOMPLISHMENTS:  The  PI  is  reminded  that  the  recipient  organization  is  required  to 
obtain  prior  written  approval  from  the  awarding  agency  Grants  Officer  whenever  there  are 
significant  changes  in  the  project  or  its  direction. 

What  were  the  major  goals  of  the  project? 

List  the  major  goals  of  the  project  as  stated  in  the  approved  SOW.  If  the  application  listed 
milestones/target  dates  for  important  activities  or  phases  of  the  project,  identify  these  dates  and 
show  actual  completion  dates  or  the  percentage  of  completion. 


End  of  Year  1:  Select  targeted  clinical  processes  and  associated  diagnostic  or  treatment 
procedures  that  will  serve  as  the  subject  matter  for  this  work;  Select,  refine  or  develop  valid 
and  reliable  metrics,  drawn  from  routinely  collected  EHR  data  to  quantify  variability  in  skill 
maintenance  and  retention  within  and  between  participating  physicians  regarding  the  targeted 
conditions. 

End  of  Year  2:  Perform  task  analysis  of  the  knowledge,  skills  and  proficiencies  needed  for 
optimal  management  of  the  targeted  clinical  processes;  Refine,  develop  and  validate 
measurement  tools  that  permit  reliable  and  valid  quantification  of  mechanisms,  processes  and 
characteristics  that  are  expected  to  mediate  or  moderate  the  observed  degree  of  clinical  skill 
maintenance  and  retention;  Determine  the  extent  to  which  clinical  skill  maintenance  or  decay 
are  associated  with  key  outcomes  of  children’s  health  care. 

End  of  Year  3:  Develop  and  test  alternative  quantitative  models  that  evaluate  the  separate  and 
interactive  associations  of  variables  at  the  level  of  demographic  characteristics  of  physicians 
and  patients,  individual  characteristics  of  physicians  and  EHR  data  as  predictors  of  the 
maintenance  of  clinical  skill  proficiency;  Plan  and  implement  a  randomized  controlled  trial  of 
educational  or  other  interventions  derived  from  the  above  model  to  enhance  the  maintenance 
and  retention  of  clinical  skills  in  a  large  sample  of  practicing  physicians;  Disseminate  project 
results  in  the  form  of  periodic  required  progress  reports  to  the  sponsor  and  the  submission 
of  abstracts,  conference  presentations  and  journal  articles  reporting  the  research  findings. 
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What  was  accomplished  under  these  goals? 

For  this  reporting  period  describe:  1 )  major  activities;  2)  specific  objectives;  3)  significant 
results  or  key  outcomes ,  including  major  findings ,  developments ,  or  conclusions  ( both  positive 
and  negative );  and/or  4)  other  achievements.  Include  a  discussion  of  stated  goals  not  met. 
Description  shall  include  pertinent  data  and  graphs  in  sufficient  detail  to  explain  any  significant 
results  achieved.  A  succinct  description  of  the  methodology  used  shall  be  provided.  As  the 
project  progresses  to  completion ,  the  emphasis  in  reporting  in  this  section  should  shift  from 
reporting  activities  to  reporting  accomplishments. 


•  We  have  completed  the  specification  of  9  targeted  clinical  problems  that  met  specific  a-priori 
selection  criteria  and  have  collected  massive  amounts  of  EHR  data  to  ensure  that  we  are 
examining  clinical  practice  dimensions  that  may  be  susceptible  to  decay  with  increasing  time 
since  prior  exposure  to  that  problem. 

•  We  have  developed  the  A-t  statistic  to  quantify  clinicians’  time  since  prior  exposure  to  the 
clinical  problem  of  interest. 

•  We  have  completed  structured  task  analysis  interviews  of  multiple  subject  matter  experts  and 
converted  the  findings  into  a  collection  of  EHR  data  elements  that  can  be  extracted  to  index  the 
degree  to  which  a  given  clinical  encounter  reflected  optimal  or  suboptimal  care  vis  a  vis  those 
elements  of  care. 

•  We  have  explored  substantial  2013  retrospective  EHR  data  relative  to  each  of  the  targeted 
clinical  problems  to  determine  which  elements  of  optimal  care  are  most  prone  to  decay  with 
increasing  values  of  the  A-t  statistic. 

•  In  response  to  questions  raised  in  response  to  our  presentations  at  In-Progress  Reviews  of  our 
work  in  August  2014  and  August,  2015,  we  have  explored  alternative  methods  of  multivariate 
modeling  of  the  relationships  between  skill  decay  and  frequency  of  exposure  and  we  continue  to 
consult  regularly  with  our  project  statistician  about  other  conceptual  and  analytic  approaches  that 
are  available  to  us.  We  continue  to  work  on  methods  of  discriminating  skill  decay  from  simple 
non-adherence  to  practice  guidelines  among  physicians  and  have  developed  several  methods  that 
we  believe  enable  us  to  discriminate  among  these  processes  validly. 

•  We  have  presented  and  published  papers  reporting  the  above  work  at  the  Military  Operations 
Research  Society,  the  Interservice/Industry  Training,  Simulation,  and  Education  Conference,  and 
the  American  Medical  Informatics  Association  and  additional  papers  are  in  preparation. 

•  We  have  initiated  recruitment  of  physicians  for  an  influenza  immunization  intervention  trial  and 
we  have  additional  intervention  trials  in  preparation  for  supracondylar  fracture,  concussion  and 
gastro-esophageal  reflux  disease.  These  will  be  of  similar  study  design  consisting  of  a  2X2 
randomized  treatments  design  manipulating  independent  variables  consisting  of  comparing  no 
intervention  with  computerized  clinical  decision  supports,  web-based  instructional  modules,  or 
their  combination  on  quality  of  care  metrics  validated  for  each  targeted  clinical  problem.  We 
expect  to  complete  several  such  intervention  trials  during  Year  3. 


What  opportunities  for  training  and  professional  development  has  the  project  provided? 

If  the  project  was  not  intended  to  provide  training  and  professional  development  opportunities  or 
there  is  nothing  significant  to  report  during  this  reporting  period,  state  “ Nothing  to  Report.  ” 

Describe  opportunities  for  training  and  professional  development  provided  to  anyone  who 
worked  on  the  project  or  anyone  who  was  involved  in  the  activities  supported  by  the  project. 
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“Training”  activities  are  those  in  which  individuals  with  advanced  professional  skills  and 
experience  assist  others  in  attaining  greater  proficiency.  Training  activities  may  include,  for 
example,  courses  or  one-on-one  work  with  a  mentor.  “Professional  development  ”  activities 
result  in  increased  knowledge  or  skill  in  one ’s  area  of  expertise  and  may  include  workshops, 
conferences,  seminars,  study  groups,  and  individual  study.  Include  participation  in  conferences, 
workshops,  and  seminars  not  listed  under  major  activities. 


Nothing  to  report.  This  project  was  not  designed  to  provide  formal  training  experiences  to 
anyone,  but  all  members  of  the  team  have  benefitted  from  the  trans-disciplinary  nature  of  the 
group’s  collaborative  interactions. 


How  were  the  results  disseminated  to  communities  of  interest? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  the  results  were  disseminated  to  communities  of  interest.  Include  any  outreach 
activities  that  were  undertaken  to  reach  members  of  communities  who  are  not  usually  aware  of 
these  project  activities,  for  the  purpose  of  enhancing  public  understanding  and  increasing 

These  papers  have  been  presented: 

•  Werk,  L,  Diaz,  MC,  Ingraham,  L,  Crutchfield,  J,  Franciosi,  J,  Wysocki,  T.  Structured 
development  of  interventions  to  improve  physician  knowledge  retention.  Interservice/Industry 
Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2015 

•  Crutchfield,  J,  Ingraham,  L,  Wysocki,  T.  Validation  of  cognitive  performance  measurements  to 
map  physician  declarative  knowledge  in  practice.  Presented  at  the  Military  Operations 
Research  Society,  Alexandria,  VA,  2015. 

•  Wysocki,  T.,  Diaz,  MC,  Werk,  L,  Franciosi,  J,  Crutchfield,  J.  Use  of  an  Electronic  Health 
Record  as  a  Research  Tool:  Frequency  of  Exposure  to  Targeted  Medical  Conditions  and  Health 
Care  Providers’  Clinical  Proficiency.  Presented  at  the  American  Medical  Informatics 
Association,  San  Francisco,  2015.  - 

This  paper  is  under  editorial  review: 

•  Wysocki,  T,  Diaz,  MC,  Werk,  L,  Franciosi,  J,  Crutchfield,  J.  Use  of  an  Electronic  Health 
Record  as  a  Research  Tool:  Frequency  of  Exposure  to  Targeted  Clinical  Problems  and  Health 
Care  Providers’  Clinical  Proficiency.  Submitted  to  the  Journal  of  the  American  Medical 
Informatics  Association. 


What  do  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals? 

Describe  briefly  what  you  plan  to  do  during  the  next  reporting  period  to  accomplish  the  goals 
and  objectives. 

We  will  continue  to  refine  the  multivariate  models  that  we  have  been  evaluating.  We  will 
apply  and  test  those  models  in  intervention  trials  for  as  many  of  the  targeted  clinical 
problems  as  possible  during  Year  3.  We  have  one  trial  in  progress  (influenza  immunization 
and  several  others  almost  ready  for  IRB  submission  (gastroesophageal  reflux  disease; 
supracondylar  fracture;  concussion). 
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4.  IMPACT:  Describe  distinctive  contributions,  major  accomplishments,  innovations,  successes,  or 
any  change  in  practice  or  behavior  that  has  come  about  as  a  result  of  the  project  relative  to: 

What  was  the  impact  on  the  development  of  the  principal  discipline(s)  of  the  project? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  findings,  results,  techniques  that  were  developed  or  extended,  or  other  products 
from  the  project  made  an  impact  or  are  likely  to  make  an  impact  on  the  base  of  knowledge, 
theory,  and  research  in  the  principal  disciplinary  field(s)  of  the  project.  Summarize  using 
language  that  an  intelligent  lay  audience  can  understand  (Scientific  American  style). 


The  impact  of  this  project  will  be  most  evident  on  the  field  of  medical  informatics  to  the  extent 
that  it  may  offer  an  excellent  illustration  of  the  integration  of  EHR  data  collection  capabilities 
with  other  data  sources  to  identify  circumstances  in  which  frequency  of  exposure  to  targeted 
clinical  problems  is  predictive  of  decay  in  physicians’  clinical  decision  making  skills  that  may 
affect  the  outcomes,  safety,  cost,  and  efficiency  of  delivered  care.  The  impact  of  this  project  will 
be  most  evident  on  the  field  of  medical  informatics  to  the  extent  that  it  may  offer  an  excellent 
illustration  of  the  integration  of  EHR  data  collection  capabilities  with  other  data  sources  to 
identify  circumstances  in  which  frequency  of  exposure  to  targeted  clinical  problems  is  predictive 
of  decay  in  physicians’  clinical  decision  making  skills  that  may  affect  the  outcomes,  safety,  cost, 
and  efficiency  of  delivered  care. 

What  was  the  impact  on  other  disciplines? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  the  findings,  results,  or  techniques  that  were  developed  or  improved,  or  other 
products  from  the  project  made  an  impact  or  are  likely  to  make  an  impact  on  other  disciplines. 


The  project  could  have  some  impact  on  other  disciplines  that  study  mechanisms  that  affect 
optimal  skill  performance  among  highly  trained  experts  in  any  field  in  which  real  time 
performance  is  routinely  measured  electronically. 


What  was  the  impact  on  technology  transfer? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ’’ 

Describe  ways  in  which  the  project  made  an  impact,  or  is  likely  to  make  an  impact,  on 
commercial  technology  or  public  use,  including: 

•  transfer  of  results  to  entities  in  government  or  industry; 

•  instances  where  the  research  has  led  to  the  initiation  of  a  start-up  company;  or 

•  adoption  of  new  practices. 


Nothing  to  Report 


What  was  the  impact  on  society  beyond  science  and  technology? 
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If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  how  results  from  the  project  made  an  impact,  or  are  likely  to  make  an  impact,  beyond 
the  bounds  of  science,  engineering,  and  the  academic  world  on  areas  such  as: 

•  improving  public  knowledge,  attitudes,  skills,  and  abilities; 

•  changing  behavior,  practices,  decision  making,  policies  (including  regulatory  policies), 
or  social  actions;  or 

•  improving  social,  economic,  civic,  or  environmental  conditions. 


To  the  extent  that  recent  increases  in  health  care  costs  are  unsustainable,  the  project  could 
contribute  to  reduction  of  health  care  costs  and  improvement  of  health  care  efficiency  if  it 
leads  to  validated  methods  to  assess  and  then  improve  decay  in  physicians’  optimal  clinical 
decision  making.  This  could  lead  to  reductions  in  health  care  waste  such  as  unnecessary 
laboratory  tests,  subspecialty  consultations,  procedures,  etc.,  while  also  potentially  reducing 
costly  medical  errors. 


5.  CHANGES/PROBLEMS:  The  Project  Director/Principal  Investigator  (PD/PI)  is  reminded  that 
the  recipient  organization  is  required  to  obtain  prior  written  approval  from  the  awarding  agency 
Grants  Officer  whenever  there  are  significant  changes  in  the  project  or  its  direction.  If  not 
previously  reported  in  writing,  provide  the  following  additional  information  or  state,  “Nothing  to 
Report,”  if  applicable: 

Changes  in  approach  and  reasons  for  change 

Describe  any  changes  in  approach  during  the  reporting  period  and  reasons  for  these  changes. 
Remember  that  significant  changes  in  objectives  and  scope  require  prior  approval  of  the  agency. 


See  below. 


Actual  or  anticipated  problems  or  delays  and  actions  or  plans  to  resolve  them 

Describe  problems  or  delays  encountered  during  the  reporting  period  and  actions  or  plans  to 
resolve  them. 

The  project  is  considerably  under-spent  for  a  wide  variety  of  reasons  that  have  been 
explained  in  previous  quarterly  and  annual  progress  reports.  We  have  discussed  with  Mr. 

Tony  Story  our  proposal  to  utilize  the  unspent  funds  in  two  broad  ways  that  constitute 
contract  modifications.  These  include  transferring  a  portion  of  the  unspent  balance  of  our 
award  to  Lockheed-Martin  subcontract  in  view  of  their  larger  than  expected  role  in  the 
project  and  our  intent  to  utilize  their  simulation  training  capabilities  in  intervention  trials 
for  idiopathic  scoliosis  and  supracondylar  fracture  in  Year  3.  Second,  we  wish  to  propose 
an  extension  of  the  ending  date  of  our  contract  from  September  14,  2016  to  March  14, 

2017.  We  are  submitting  separate  written  requests  for  each  of  these  proposals  as  instructed 
by  Mr.  Story. 


Changes  that  had  a  significant  impact  on  expenditures 
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Describe  changes  during  the  reporting  period  that  may  have  had  a  significant  impact  on 
expenditures,  for  example,  delays  in  hiring  staff  or  favorable  developments  that  enable  meeting 
objectives  at  less  cost  than  anticipated. 


These  changes  have  impacted  expenditures: 

1 .  We  have  been  able  to  achieve  the  stated  objectives  of  the  contract  to  this  point  with  less 
personnel  time  and  effort  than  was  projected. 

2.  We  have  intentionally  held  back  on  project  expenditures  anticipating  that  Year  3,  with  work 
consisting  almost  exclusively  of  intervention  trials,  would  be  by  far  the  most  expensive  of 
the  three  project  years. 

3.  Dr.  David  Milov,  who  was  to  occupy  a  key  20%  FTE  role  in  the  project,  experienced  a 
catastrophic  illness  more  than  1 8  months  ago  and  he  has  yet  to  return  to  work.  Nemours  has 
not  replaced  him  as  Director  of  Medical  Informatics  because  his  eventual  return  has  been 
ambiguous. 


Significant  changes  in  use  or  care  of  human  subjects,  vertebrate  animals,  biohazards, 
and/or  select  agents 

Describe  significant  deviations,  unexpected  outcomes,  or  changes  in  approved  protocols  for  the 
use  or  care  of  human  subjects,  vertebrate  animals,  biohazards,  and/or  select  agents  during  the 
reporting  period.  If  required,  were  these  changes  approved  by  the  applicable  institution 
committee  (or  equivalent )  and  reported  to  the  agency?  Also  specify  the  applicable  Institutional 
Review  Board/Institutional  Animal  Care  and  Use  Committee  approval  dates. 

Significant  changes  in  use  or  care  of  human  subjects 


Nothing  to  Report 


Significant  changes  in  use  or  care  of  vertebrate  animals. 


Nothing  to  Report 


Significant  changes  in  use  of  biohazards  and/or  select  agents 


Nothing  to  Report 
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6.  PRODUCTS:  List  any  products  resulting  from  the  project  during  the  reporting  period.  If 
there  is  nothing  to  report  under  a  particular  item,  state  “Nothing  to  Report.” 

•  Publications,  conference  papers,  and  presentations 

Report  only  the  major  publication(s)  resulting  from  the  work  under  this  award. 

Journal  publications.  List  peer-reviewed  articles  or  papers  appearing  in  scientific, 
technical,  or  professional  journals.  Identify  for  each  publication:  Author(s);  title; 
journal;  volume:  year;  page  numbers;  status  of  publication  (published;  accepted, 
awaiting  publication;  submitted,  under  review;  other);  acknowledgement  of  federal 
support  (yes/no). 


These  papers  have  been  presented: 

•  Werk,  L,  Diaz,  MC,  Ingraham,  L,  Crutchfield,  J,  Franciosi,  J,  Wysocki,  T.  Structured 
development  of  interventions  to  improve  physician  knowledge  retention. 
Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2015 

•  Crutchfield,  J,  Ingraham,  L,  Wysocki,  T.  Validation  of  cognitive  performance 
measurements  to  map  physician  declarative  knowledge  in  practice.  Presented  at  the 
Military  Operations  Research  Society,  Alexandria,  VA,  2015. 

•  Wysocki,  T.,  Diaz,  MC,  Werk,  L,  Franciosi,  J,  Crutchfield,  J.  Use  of  an  Electronic 
Health  Record  as  a  Research  Tool:  Frequency  of  Exposure  to  Targeted  Medical 
Conditions  and  Health  Care  Providers’  Clinical  Proficiency.  Presented  at  the 
American  Medical  Informatics  Association,  San  Francisco,  2015.  - 

This  paper  is  under  editorial  review: 

•  Wysocki,  T,  Diaz,  MC,  Werk,  L,  Franciosi,  J,  Crutchfield,  J.  Use  of  an  Electronic 
Health  Record  as  a  Research  Tool:  Frequency  of  Exposure  to  Targeted  Clinical 
Problems  and  Health  Care  Providers’  Clinical  Proficiency.  Submitted  to  the  Journal  of 
the  American  Medical  Informatics  Association. 


Books  or  other  non-periodical,  one-time  publications.  Report  any  book,  monograph, 
dissertation,  abstract,  or  the  like  published  as  or  in  a  separate  publication,  rather  than  a 
periodical  or  series.  Include  any  significant  publication  in  the  proceedings  of  a  one-time 
conference  or  in  the  report  of  a  one-time  study,  commission,  or  the  like.  Identify  for  each 
one-time  publication:  Author(s);  title;  editor;  title  of  collection,  if  applicable; 
bibliographic  information;  year;  type  of  publication  (e.g.,  book,  thesis  or  dissertation); 
status  of  publication  ( published ;  accepted,  awaiting  publication;  submitted,  under 
review;  other);  acknowledgement  of  federal  support  (yes/no). 


Nothing  to  Report 
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Other  publications,  conference  papers,  and  presentations.  Identify  any  other 
publications,  conference  papers  and/or  presentations  not  reported  above.  Specify  the 
status  of  the  publication  as  noted  above.  List  presentations  made  during  the  last  year 
(international,  national,  local  societies,  military  meetings,  etc.).  Use  an  asterisk  (*)  if 
presentation  produced  a  manuscript. 


•  Werk,  L,  Diaz,  MC,  Ingraham,  L,  Crutchfield,  J,  Franciosi,  J,  Wysocki,  T. 
Structured  development  of  interventions  to  improve  physician  knowledge 
retention.  Interservice/Industry  Training,  Simulation,  and  Education  Conference 
(I/ITSEC)  2015* 

•  Wysocki,  T.,  Diaz,  MC,  Werk,  L,  Franciosi,  J,  Crutchfield,  J.  Use  of  an  Electronic 
Health  Record  as  a  Research  Tool:  Frequency  of  Exposure  to  Targeted  Medical 
Conditions  and  Health  Care  Providers’  Clinical  Proficiency.  Presented  at  the 
American  Medical  Informatics  Association,  San  Francisco,  2015.  * 

*Full  journal  articles  resulted  from  both  of  the  above  presentations. 

•  Website(s)  or  other  Internet  site(s) 

List  the  URL  for  any  Internet  site(s)  that  disseminates  the  results  of  the  research 
activities.  A  short  description  of  each  site  should  be  provided.  It  is  not  necessary  to 
include  the  publications  already  specified  above  in  this  section. 


Nothing  to  Report 


•  Technologies  or  techniques 

Identify  technologies  or  techniques  that  resulted  from  the  research  activities.  In  addition 
to  a  description  of  the  technologies  or  techniques,  describe  how  they  will  be  shared. 


Nothing  to  Report 


•  Inventions,  patent  applications,  and/or  licenses 

Identify  inventions,  patent  applications  with  date,  and/or  licenses  that  have  resulted  from 
the  research.  State  whether  an  application  is  provisional  or  non-provisional  and  indicate 
the  application  number.  Submission  of  this  information  as  part  of  an  interim  research 
performance  progress  report  is  not  a  substitute  for  any  other  invention  reporting 
required  under  the  terms  and  conditions  of  an  award. 


Nothing  to  Report 


•  Other  Products 

Identify  any  other  reportable  outcomes  that  were  developed  under  this  project. 
Reportable  outcomes  are  defined  as  a  research  result  that  is  or  relates  to  a  product, 
scientific  advance,  or  research  tool  that  makes  a  meaningful  contribution  toward  the 
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understanding,  prevention,  diagnosis,  prognosis,  treatment,  and/or  rehabilitation  of  a 
disease,  injury  or  condition,  or  to  improve  the  quality  of  life.  Examples  include: 

•  data  or  databases; 

•  biospecimen  collections; 

•  audio  or  video  products; 

•  software; 

•  models; 

•  educational  aids  or  curricula; 

•  instruments  or  equipment; 

•  research  material  (e.g.,  Germplasm;  cell  lines,  DNA  probes,  animal  models); 

•  clinical  interventions; 

•  new  business  creation;  and 

•  other. 


•  Development  and  validation  of  the  A-t  statistic  to  measure  duration  of  time  elapsing 
since  the  prior  clinical  exposure  to  a  specific  targeted  clinical  problem  and 
understanding  of  its  distributional  characteristics. 

•  Development  of  quality  of  care  metrics  for  the  9  targeted  clinical  problems  based  on 
structured  interviews  of  subject  matter  experts,  review  of  published  guidelines  for 
evidence-based  clinical  practice,  and  evaluation  of  feasibility  of  HER  data  extraction  for 
each  such  metric. 

•  Development  of  a  framework  for  systematic  development  of  behavioral,  education  and 
clinical  decision  support  interventions  matched  to  learner  needs  based  on  observations 
of  decay  in  quality  of  care  metrics  as  a  function  of  increasing  values  of  the  A-t  statistic. 


7.  PARTICIPANTS  &  OTHER  COLLABORATING  ORGANIZATIONS 
What  individuals  have  worked  on  the  project? 

Provide  the  following  information  for:  (1)  PDs/PIs;  and  (2)  each  person  who  has  worked  at  least 
one  person  month  per  year  on  the  project  during  the  reporting  period,  regardless  of  the  source 
of  compensation  (a  person  month  equals  approximately  160  hours  of  effort).  If  information  is 
unchanged  from  a  previous  submission,  provide  the  name  only  and  indicate  “no  change.  ” 

Example : 

Name:  Mary  Smith 

Project  Role:  Graduate  Student 

Researcher  Identifier  (e.g.  ORCID  ID):  1234567 
Nearest  person  month  worked:  5 

Contribution  to  Project:  Ms.  Smith  has  performed  work  in  the  area  of 

combined  error-control  and  constrained  coding. 
Funding  Support:  The  Ford  Foundation  ( Complete  only  if  the  funding 

support  is  provided  from  other  than  this  award). 
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Personnel  Name 

Project  Role 

Researcher 

Identifier 

Person 

Month 

Worked 

Contribution  to  Project 

Funding 
support  from 
other  than 
this  award 

Tim  Wysocki,  PhD 

PI 

http  ://orcid.org/00 

00-0003-4099- 

4639 

3.68 

Dr.  Wysocki  is  PI,  responsible 
for  conduct  of  the  project  and 
achieving  the  specific  aims  and 
statement  of  work  on  schedule 
within  budget 

N/A 

M.  C.  Diaz,  MD 

Co-I 

N/A 

0.98 

Dr.  Diaz  is  a  member  of  the 
Targeting  Clinical  Problems 
Working  Group  and  the 
Interventions  Working  Group. 

N/A 

J.  Franciosi,  MD 

Co-I 

N/A 

0.59 

Dr.  Franciosi  is  a  pediatric 
gastroenterologist  and  a  member 
of  the  Model  Building  and 
Analysis  Working  Group. 

N/A 

S.  Lawless,  MD,  MBA 

Co-I 

N/A 

0.03 

Dr.  Lawless  is  Vice  President  for 
Quality  and  Safety  and  a 
member  of  the  Steering 
Committee.  He  facilitates  access 
to  data  from  Nemours  Data 
Warehouse,  the  Physician 
Credentialing  Database  and 
other  sources. 

N/A 

D.  Milov,  MD 

Co-I 

N/A 

0.00 

Dr.  Milov  is  Chief  of  Medical 
Informatics  and  a  member  of  the 
Steering  Committee,  Chair  of 
the  Targeting  Clinical  Problems 
Working  Group  and  member  of 
the  Model  Building  and  Analysis 
Working  Group. 

N/A 

L.  Werk,  MD,  MPH 

Co-I 

N/A 

2.38 

Dr.  Werk  is  Chief  of  General 
Pediatrics  at  Nemours 

Children’s  Hospital  in  Orlando, 
a  member  of  the  Steering 
Committee  and  Chair  of  the 
Interventions  Working  Group. 

N/A 

J.  Hossain,  PhD 

Stats. 

N/A 

0.97 

Dr.  Hossain  is  a  Ph.D. 

Statistician  and  a  member  of  the 
Targeting  Clinical  Problems  and 
Model  Building  and  Analysis 
Working  Groups. 

N/A 

A. Taylor,  MA 

Data  Mgr. 

N/A 

2.71 

Mrs.  Taylor  has  developed 
agenda  for  conference  calls  and 
meetings,  drafted  and  posted 
minutes,  and  assisted  with  data 
extraction 

N/A 

J.  Miller 

EMR  Spec. 

NA 

1.52 

Mr.  Miller  coordinates 
extraction  of  data  from  Nemours 
EMR  Data  Warehouse 

N/A 
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R.  Villanueva 

EMR  Spec. 

N/A 

1.98 

Mr.  Villanueva  evaluates  and 
adjusts  for  nuances  in  clinical 
service  documentation  across 
physicians  and  Nemours  sites. 

N/A 

E.  Antico 

EMR  Spec 

N/A 

7.43 

Mrs.  Antico  coordinates 
extraction  of  data  from  Nemours 
EMR  Data  Warehouse 

N/A 

D.  Kemp 

Proj.  Mgr. 

N/A 

1.81 

Mr.  Kemp  is  a  Certified  Project 
Manager  whose  time  and  effort 
are  donated  in-kind  by  Nemours 

The  Nemours 
Foundation 

J.  Dent 

Proj.  Mgr. 

N/A 

1.81 

Ms.  Dent  is  a  Certified  Project 
Manager  whose  time  and  effort 
are  donated  in-kind  by  Nemours 

The  Nemours 
Foundation 

J.  Crutchfield,  PhD 

PI  (LM) 

N/A 

9.39 

Dr.  Crutchfield  is  a  Sociologist 
and  expert  in  quantitative 
modeling  of  maintenance  and 
decay  of  human  performance 
and  cognition. 

N/A 

L.  Ingraham,  MSCE 

Sr.  Project 

Engineer 

(LM) 

N/A 

5.90 

Ms.  Ingraham  is  the  Project 
Engineer  who  is  an  expert  at  task 
analysis  and  matching  of 
educational  and  decision  support 
interventions  to  characteristics 
of  tasks,  workers  and 
workplaces. 

N/A 

Y.  Marks,  PhD 

Instructional 
Designer  LM) 

N/A 

2.18 

Dr.  Marks  is  an  expert  at 
conducting  interviews  of  subject 
matter  experts  to  identify 
intervention  targets  that  can  lead 
to  performance  improvement. 

N/A 

T.  Henfield 

Instructional 

Designer 

(LM) 

N/A 

0.38 

Ms.  Henfield  is  an  expert  at 
conducting  interviews  of  subject 
matter  experts  to  identify 
intervention  targets  that  can  lead 
to  performance  improvement. 

N/A 

R.  Marra 

Instructional 

Designer 

(LM) 

N/A 

0.35 

Mr.  Marra  is  an  engineer  who 
specializes  in  designing 
interventions  matched  to 
features  of  tasks,  workers  and 
workplaces. 

N/A 

Has  there  been  a  change  in  the  active  other  support  of  the  PD/PI(s)  or  senior/key  personnel 
since  the  last  reporting  period? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

If  the  active  support  has  changed  for  the  PD/PI(s)  or  senior/key  personnel,  then  describe  what 
the  change  has  been.  Changes  may  occur,  for  example,  if  a  previously  active  grant  has  closed 
and/or  if  a  previously  pending  grant  is  now  active.  Annotate  this  information  so  it  is  clear  what 
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has  changed  from  the  previous  submission.  Submission  of  other  support  information  is  not 
necessary  for  pending  changes  or  for  changes  in  the  level  of  effort  for  active  support  reported 
previously.  The  awarding  agency  may  require  prior  written  approval  if  a  change  in  active  other 
support  significantly  impacts  the  effort  on  the  project  that  is  the  subject  of  the  project  report. 


Nothing  to  Report 


What  other  organizations  were  involved  as  partners? 

If  there  is  nothing  significant  to  report  during  this  reporting  period,  state  “Nothing  to  Report.  ” 

Describe  partner  organizations  -  academic  institutions,  other  nonprofits,  industrial  or 
commercial  firms,  state  or  local  governments,  schools  or  school  systems,  or  other  organizations 
(foreign  or  domestic )  -  that  were  involved  with  the  project.  Partner  organizations  may  have 
provided  financial  or  in-kind  support,  supplied  facilities  or  equipment,  collaborated  in  the 
research,  exchanged  personnel,  or  otherwise  contributed. 

Provide  the  following  information  for  each  partnership: 

Organization  Name: 

Location  of  Organization:  (if  foreign  location  list  country ) 

Partner 's  contribution  to  the  project  (identify  one  or  more ) 

•  Financial  support; 

•  In-kind  support  (e.g.,  partner  makes  software,  computers,  equipment,  etc., 
available  to  project  staff); 

•  Facilities  (e.g.,  project  staff  use  the  partner ’s  facilities  for  project  activities); 

•  Collaboration  (e.g.,  partner ’s  staff  work  with  project  staff  on  the  project); 

•  Personnel  exchanges  (e.g.,  project  staff  and/or  partner ’s  staff  use  each  other ’s  facilities, 
work  at  each  other ’s  site);  and 

•  Other. 


The  project  includes  a  substantial  Subaward  with  the  Lockheed  Martin  Corporation  (LMCO)  Mission 
Systems  and  Training  Division  located  in  Orlando  Florida.  This  team’s  expertise  is  a  major  asset  to  the 
project  and  consists  of  extensive  experience  in  studying  and  remediating  skill  decay  among  military  and 
commercial  pilots  as  well  as  among  health  care  professionals.  They  contribute  their  expertise  to  the  study  by 
conducting  and  analyzing  detailed  task  analysis  interviews  of  Subject  Matter  Experts  (SMEs)  in  the  clinical 
management  of  our  targeted  clinical  problems;  working  with  Mr.  Miller  and  Mr.  Villanueva  to  determine 
how  critical  elements  of  the  care  process  for  each  clinical  problem  can  be  extracted  from  the  Nemours  EMR 
Data  Warehouse,  identifying  critical  worker  and  workplace  variables  that  can  be  measured,  and  finally 
developing  quantitative  models  for  prediction  of  decay  in  selected  clinical  skills  and  decisions.  Several 
documents  that  demonstrate  fulfillment  of  the  LMCO  Statement  of  Work  for  Year  2  under  this  Subaward 
Agreement  are  included  in  the  Appendix  to  this  report. 


8.  SPECIAL  REPORTING  REQUIREMENTS 
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COLLABORATIVE  AWARDS:  For  collaborative  awards,  independent  reports  are  required 
from  BOTH  the  Initiating  PI  and  the  Collaborating/Partnering  PI.  A  duplicative  report  is 
acceptable;  however,  tasks  shall  be  clearly  marked  with  the  responsible  PI  and  research  site.  A 
report  shall  be  submitted  to  https://ers.amedd.army.mil  for  each  unique  award. 

QUAD  CHARTS:  If  applicable,  the  Quad  Chart  (available  on  https://www.usamraa.army.mil) 
should  be  updated  and  submitted  with  attachments. 
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Maintenance  of  Health  Care  Providers’  Clinical  Proficiency: 
Transdisciplinary  Analysis,  Modeling  and  Intervention 

Log  Number:  12362007 
Award  Number:  W81XWH-13-1-0316 


DMRDP 


PI:  Tim  Wysocki,  Ph.D. 


Organization:  The  Nemours  Foundation  Award  Amount:  $3,936,682 


Problem,  Hypothesis  and  Military  Relevance 

•  Military  physicians  may  demonstrate  skill  degradation 
after  deployment  upon  their  return  to  domestic  practices 

•  Skill  degradation  may  impede  efficiency  and  safety  of  care 

•  DOD  delivers  care  to  many  healthy  adolescent  and  young 
adult  enlisted  men  and  to  military  dependents,  justifying 
research  on  skill  degradation  in  pediatrics 


Proposed  Solution 


Nemours  Locations 

DELAWARE:  Allred  I .  duP ont  Hospital 
for  C  hildren,  Wilm  ington;  Specialty  P  ediatrics 
Wilmington;  Cardiac  Center;  Wilm  ington;  Primary 
Care  Pediatrics:  Wilm  ington11Milford1  D  over,,N  ewark , 
Middletown,  Sea  ford 

PENNSYLVANIA:  Specialty  P ediatrics:  Lancaster, 
College vi lie,  Newtown  Square,  P  hiladelphia;  P  rim  ary 
Care  Pediatrics:  Lankenau ,  P  hiladelphia 

NEW  JERSEY:  Specialty  Pediatrics:  Egg  Harbor 
T ownship ,  Vineland,  Voorhees 

FLORIDA:  Nem  ours  C  hildren's  Hospital,  Orlando 
(opening  late  2012);  Specialty  Pediatrics:  Orlando, 
Jacksonville ,  P  ensacola ,  Orange  P  ark ,  D  estin  .Lake 
Mary,  Viera 


Timeline  and  Total  Cost 


Objective  1 :  We  will  examine  frequency  of  exposure  to 
certain  targeted  medical  conditions  as  a  surrogate  context 
for  studying  skill  degradation  among  pediatric  health  care 
providers 

Objective  2:  Develop  and  validate  quality  metrics,  test 
alternative  conceptual  models  and  develop  and  test 
appropriate  intervention  strategies  to  prevent/minimize 
skill  degradation 


Activities 

FY13 

FY14 

FY15 

FY16 

Assemble  research  team;  plan  EMR  and  data 
warehouse  inquires  to  select  targeted  medical 
conditions.  Catalog  and  evaluate  existing  applicable 
quality  metrics. 

Develop  quantitative  conceptual/explanatory  models 
of  clinical  skill  decay  among  pediatric  health  care 
providers.  Plan  educational  interventions  and  EMR 
decision  support  tools  to  prevent  clinical  skill  decay. 

Conduct  randomized  controlled  trials  of 
interventions;  final  statistical  analysis;  disseminate 
final  project  results,  prepare  subsequent  grant 
applications  to  continue  inquiry. 

Estimated  Total  Budget  ($3.9M) 

$0.5M 

$1.4M 

$1.3M 

$0.7M 

Updated:  12-OCT-2015 


9.  APPENDICES:  Attach  all  appendices  that  contain  information  that  supplements,  clarifies  or 
supports  the  text.  Examples  include  original  copies  of  journal  articles,  reprints  of  manuscripts 
and  abstracts,  a  curriculum  vitae,  patent  applications,  study  questionnaires,  and  surveys,  etc. 
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Healthcare  Skill  Maintenance  Model 
PERFORMANCE  METRIC  CONSTRUCTION 

YEAR  2  FORMAL  REPORT 


Prepared  by: 

Nemours,  A  Childrens  Health  System 


and 

LOCKHEED  MAR  I  N 

Mission  Systems  &  Training 


Orlando,  Florida  32825 
March  24,  2015 
For 

U.S.  Army,  Congressionally  Directed  Medical  Research  Program 


DISTRIBUTION  STATEMENT  D:  Distribution  Authorized  to  the  Department  of  Defense  and  US  DoD  Contractors  only 
(Critical  Technology),  30  January  2014.  Other  requests  shall  be  referenced  to  U.S.  Army,  Telemedicine  and  Advanced 
Technology  Research  Center. 

DESTRUCTION  NOTICE:  Destroy  by  any  method  that  will  prevent  disclosure  of  contents  or  reconstruction  of  the 
document. 

WARNING:  This  document  contains  technical  data  whose  export  is  restricted  by  the  Arms  Export  Control  Act  (Title  22, 
U.S.C.  2571  et  seq.)  or  Executive  Order  12470.  Violation  of  export-control  laws  is  subject  to  criminal  penalties. 
Dissemination  of  this  document  is  controlled  by  DoD  Directive  5230.25  and  AFI  61-204. 


Congressionally  Directed  Medical  Research  Programs 

CDMR# 

Department  of  Defense 


For  internal  team  use  only  -  Do  Not  Distribute 


1.  CHANGE  HISTORY 


DATE 

SECTION 

DESCRIPTION  of  CHANGE 

03/24/2015 

- 

Initial  version 

ii 


For  internal  team  use  only  -  Do  not  redistribute 


2.  EXECUTIVE  SUMMARY 


Lockheed  Martin,  in  partnership  with  Nemours  Children's  Healthcare  has  been  contracted  by  the  US 
Army's  Telemedicine  &  Advanced  Technology  Research  Center,  (TATRC)  and  the  Congressionally 
Directed  Medical  Research  Program  to  investigate  the  phenomenon  of  knowledge  decay  and  develop 
evidence  based  guidelines  for  determining  retraining  and  other  intervention  schedules.  The  research 
committees  select  targeted  clinical  conditions  for  performance  evaluation  and  analysis  of  knowledge 
decay.  Measurement  of  performance  in  each  targeted  clinical  condition  will  be  developed  using  existing 
electronic  measurement  records  (EMR)  as  a  data  source.  This  paper  describes  the  steps  used  to  develop 
a  performance  metric: 

•  Identify  key  knowledge 

•  Map  knowledge  to  observations  from  EMR 

•  Select  knowledge  that  decays 

•  Construct  performance  metric  utility  function. 


For  internal  team  use  only  -  Do  not  redistribute 


3.  TABLE  OF  CONTENTS 
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5.  PURPOSE 


This  report  is  presented  as  the  deliverable  in  response  to  the  contractual  obligations  as  defined  in  the 
statement: 

During  the  first  quarter  of  Year  2,  provision  of  a  formal  written  proposal  of  the  measurement  protocol  for 
collection  of  the  data  needed  to  test  and  compare  the  validity  of  one  or  more  alternative  quantitative 
models  of  variables  associated  with  skill  decay  of  pediatric  health  care  providers. 
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6.  INTRODUCTION 

Most  complex  knowledge  is  perishable.  It  degrades  through  disuse.  Jobs  requiring  advanced  technical 
knowledge  such  as  pilots  and  physicians  are  not  immune  to  knowledge  decay.  Job  aids,  retraining  and 
refresher  training  programs  are  used  to  maintain  minimal  performance  levels,  but  determining  how 
much  retraining  is  needed  and  on  what  knowledge  has  generally  been  applied  through  educated 
guesswork. 

Lockheed  Martin,  in  partnership  with  Nemours  Children's  Healthcare  has  been  contracted  by  the  US 
Army's  Telemedicine  &  Advanced  Technology  Research  Center,  (TATRC)  and  the  Congressionally 
Directed  Medical  Research  Program  to  investigate  the  phenomenon  of  knowledge  decay  and  develop 
evidence  based  guidelines  for  determining  retraining  and  other  intervention  schedules.  The  research 
committees  select  targeted  clinical  conditions  for  performance  evaluation  and  analysis  of  knowledge 
decay.  Measures  of  performance  in  each  targeted  clinical  condition  were  developed  using  existing 
electronic  measurement  records  (EMR)  as  a  data  source.  This  paper  describes  the  four  basic  steps, 
shown  as  ovals  in  Figure  1,  used  to  develop  each  performance  metric: 

I.  Identify  key  knowledge 

II.  Map  knowledge  to  electronic  medical  records  data 

III.  Select  knowledge  that  evidences  decay 

IV.  Construct  performance  metric  utility  function. 


The  resulting  performance  metric  will  indicate  variation  as  a  result  of  time  between  performances,  and 
will  tend  to  be  higher  for  those  performers  whose  time  between  performance  is  short.  We  will  use  it  to 
explore  other  factors  that  may  affect  performance  such  as  individual  characteristics,  characteristics  of 
the  workplace  and  characteristics  of  the  performers  themselves. 


Figure  1.  Performance  Metric  Construction 
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7.  IDENTIFY  KEY  KNOWLEDGE 


CTA  analysts  interview  targeted  clinical  problem  Subject  Matter  Expert 


Each  targeted  clinical  condition  will  call  forth  from  the  physician  a  specific  set  of  clinical  knowledge.  This 
research  is  specifically 
interested  in  identifying 
declarative  knowledge  skills. 

The  process  employs  a 
cognitive  task  analyst.  A 
detailed  description,  shown 
in  Figure  2,  that  it  follows  six 
steps: 


1.  Obtain  and  analyze 
available 

documentation  on 
the  condition 

2.  Construct  a  set  of 

questions  that  will 
encourage  the 

subject  matter 

expert  to  express  his 
or  her  knowledge  of 
the  diagnosis  and 
treatment  of  the 
targeted  clinical 
condition. 

3.  Interview  the  subject 
matter  expert,  using 
the  questions. 


4.  Transcribe 

the 

interview. 

5.  Analyze 

the 

interview 

and 

additional 

materials 

and  construct  a  draft 
skill  set 

6.  Use  draft  skill  set  to 
extract 

o  Identified 
comorbid 
conditions 
o  Ordered  procedures  and  referrals 
o  Ordered  pharmaceuticals 

7.  Review  and  revise  the  skill  set  with  the  help  of  the  subject  matter  expert  in  a  short  second 
interview 


Figure  2.  Identification  of  knowledge  and  indicators  of  the  exercise  of 
that  knowledge 


Once  a  set  of  critical  skills  have  been  identified,  and  reviewed  by  the  subject  matter  expert,  we  are 
ready  to  mine  the  electronic  medical  record  data  for  those  skills. 
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8.  MAP  KEY  KNOWLEDGE  TO  ELECTRONIC  MEDICAL  RECORD  DATA 


A  search  of  the  Electronic  Medical  Record  (EMR)  data  dictionary  is  conducted  to  identify  available 
metrics.  Further  interviews  with  the  subject  matter  expert  may  be  necessary  to  extract  data  from  open- 
ended  notation  fields.  This  work  is  performed  by  Nemours  employees  since  primary  data  sources 
include  patient  identification  information. 

Initially,  no  limitation  is  placed  on  data  field  selection,  except  that  necessary  to  protect  Health 
Information  Protection  Act  (HIPA)  patient  identity  and  physician  data.  Unused  fields  or  fields  with  no 
data  are  identified  and  dropped.  If  necessary,  different  tables  are  joined  to  construct  a  base  working 
data  set.  Where  necessary,  identity  masking  techniques  are  employed,  such  as  replacing  identity 
numbers  with  random  numbers.  The  resulting  data  set  reflects  actual  medical  histories,  but  with  all 
references  to  the  patients  and  their  physicians  masked  or  redacted. 

Redacted  data  with  performance  metrics  are  extracted  and  stored  separately  for  model  analysis. 
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9.  SELECT  KNOWLEDGE  INDICATORS  THAT  EVIDENCE  PERFORMANCE  DECAY 


Some  knowledge  requirements  are  shared  by  other  clinical  conditions  or  are  of 
such  an  obvious  nature  that  under  normal  conditions,  they  do  not  evidence  decay. 

There  may  be  something  about  how  the  knowledge  was  acquired  that  may  also 
influence  retentions.  Were  these  knowledge  elements  employed  in  the  data  used 
to  construct  a  performance  metric,  they  would  mask  the  dynamic  variable  nature 
of  other  knowledge. 

Initial  attempts  at  constructing  a  performance  metric  for  the  Obesity  condition  was  a  case  in  point. 
Advising  the  patient  and/or  parents  of  the  patient  that  the  patient  needs  to  lose  weight  and  to  increase 
exercise  is  a  basic  response.  One  need  not  be  a  physician  to  know  that,  though  due  to  the  particular 
responsibility  of  the  physician  in  assisting  with  the  health  of  the  patient  it  is  a  more  urgent  matter.  We 
found  little  dynamic  variation  in  any  of  the  performance  metrics  we  attempted  to  construct.  However, 
by  removing  metrics  from  analysis  that  specifically  demonstrated  little  dynamic  variation,  we  did  find 
some  variation.  Initially  we  resisted  this  ad  hoc  response.  However,  once  we  recognized  that  there  are 
classes  of  performance  that  do  not  vary  over  time,  and  those  that  do,  we  added  an  analytical  step  to 
classify  those  metrics  prior  to  model  inclusion. 

A  dynamic  measure  of  time  (At)  was  constructed  that  represented  the  time  period  or  gap  between 
knowledge  performance  events.  This  At  is  the  time  period  we  hypothesize  to  be  responsible  for 
knowledge  decay.  Time  is  assumed  to  be  a  reasonable  proxy  for  the  unobserved  processes  that  result  in 
forgetting.  Candidate  performance  metrics  are  plotted  against  At.  Those  metrics  that  show  no  trend  or 
knowledge  decay  over  time  are  deleted  from  the  study. 
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10.  CONSTRUCT  PERFORMANCE  UTILITY  FUNCTION 


At  this  point,  for  each  targeted  clinical  condition,  we  have  identified  multiple  performance  metrics  for 
knowledge  that  evidences  decay.  As  shown  in  Figure  3,  where  the  number  of  available  indicators  is 
large,  greater  than  five,  we  combine  them  into  one  or  more  performance  utility  functions  (PUF).  Where 
the  number  of  available  indicators  is  sparse,  less  than  five,  we  may  use  the  indicators  directly.  Where 
the  available  indicators  are  few  and  dichotomous,  we  perform  a  logistic  transformation,  using  the  log- 
odds  ratio  of  the  indicator  in  the  construction  of  the  PUF. 

Computation  of  log-odds  for  dichotomous,  yes-no  variables 

Model  construction  with  a  small  number  of  dichotomous  variables  requires  a  specific  approach.  The 
arithmetic  mean  value  of  the  score  at  each  At  is  computed  for  each  healthcare  giver.  With  dichotomous 
variables,  this  will  range  from  zero  to  one.  This  can  be  interpreted  as  the  expected  value  for  the 
performance  at  that  At,  "E(f)".  The  odds  ratio  is  the  ratio  of  that  value  to  one  minus  that  value, 


m 


1  -  E(t) 


The  log  of  the  odds  ratio  is  referred  to  as  the  "log-odds  ratio"  also  known  as  the  "logistic  transform"  or 
"logit  transform," 


This  gives  us  a  value  for  use  as  a  continuous  outcome  variable  enabling  application  in  decay  model 
estimation. 

There  are  two  special  cases  that  must  be  addressed,  when  E(t)  =  1  or  E(t)  =  0.  The  former  condition  can 
occur  when  the  number  of  observations  is  low,  and  all  indicate  a  positive  (1)  performance.  In  this  case, 
the  ratio  computes  a  zero  in  the  denominator.  The  second  is  when  the  performance  is  zero,  resulting  in 
an  attempt  to  compute  the  log  of  zero.  Both  outcomes  are  mathematically  undefined.  To  compensate, 
we  add  the  value  of  0.001  to  both  the  numerator  and  the  denominator, 


This  results  in  an  approximate  continuous  range 
of  about  -1  to  12,  which  enables  analysis  without 
encountering  the  undefined  value  error. 


Computation  of  PUF  when  number  of  indicators 
greater  than  four 


When  number  of  variables  is  greater  than  four, 
we  create  performance  utility  functions  (PUF)  for 
each  of  the  identifiable  underlying  latent  metrics. 
The  variables  are  weighted  and  summed  in  a 
linear  equation  to  produce  the  PUF  value.  We 
employ  principal  component  analysis  to 
determine  these  factor  weights.  Examination  of 
the  relative  factor  weights  assists  in  identifying 
the  latent  variable  driving  the  outcomes. 


Apply  PUF 
to  Decay 


Figure  3.  Performance  utility  function  construction.  Model 
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When  the  number  of  variables  is  larger,  even  though  they  may  be  dichotomous,  they  can  be  combined 
into  a  linear  equation  using  this  approach. 

For  purposes  of  this  study,  the  knowledge  performance  metrics  either  directly  as  indicators,  or  indirectly 
as  log-odds  indictors  are  combined  into  one  or  more  performance  utility  functions.  We  will  use  a  linear 
function  for  performance,  and  this  requires  a  process  for  determining  factor  weights  for  each  of  the 
selected  metrics.  We  employ  principal  component  analysis  to  determine  these  factor  weights. 

Selection  of  max  PUF  to  differentiate  adherence  in  model  construction 

An  additional  source  of  difficulty  in  determining  a  decay  model  had  to  do  with  the  degree  of  normal 
variation  as  observed  in  our  performance  metrics.  Deviation  from  performance  guidelines,  as  measured 
through  knowledge  expression,  may  be  a  function  of  decay.  However,  it  could  also  be  the  result  of 
special  conditions  recognized  and  implemented  at  the  discretion  of  the  physician.  This  has  been  referred 
to  as  guideline  adherence  variation.  Consequently,  we  observed  substantial  variation  in  performance  at 
every  time  period.  We  also  found  that  the  maximum  values  of  the  performance  functions  often 
decreases  as  At  increases.  We  can  reasonably  assume  that  this  decrease  in  value  is  most  likely  due  to 
knowledge  decay. 
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11.  EXAMPLE  PUF  STUDIES 


The  table  excerpt  below  is  from  the  Declarative  Knowledge  Matrix  constructed  for  the  study  of  the 
Obesity  targeted  clinical  problem.  It  maps  consolidated  information  from  the  subject  matter  expert 
interview  to  the  individual  field  elements  made  available  for  analysis  by  the  Nemours  IT  team. 


ID 

Domain 

Data  Elements 

Field  Name(s) 

Notes-> 

Phase  or 
diagnosis/ 
treatment 
step 

Knowledge  identified  from 
interview  and  document  content 
analysis 

Name  from  SAS  datafile 

1.0 

General 

TCP  Description 

1.0.1 

BMI  85th  to  94th  Percentile  -  Overweight 

1.0.2 

BMI  >  95th  Percentile  -  Obese 

1.1 

Monitoring 

Physical  Description 

1.1.1 

Children  should  be  screened  by  the  primary  care  team 
for  developmental  delays  (at  the  9-,  18-,  and  either  24- 
or  30-mo  visits)  using  validated  instruments. 

1.1.2 

The  child’s  length/height  and  weight  should  be 
measured  and  documented  at  every  well-child  visit  or 
note  a  reason  for  not  doing  so. 

1.1.3 

If  a  child  is  2  y  or  older,  the  child’s  BMI  should  be 
calculated  and  documented  at  every  well-child  visit  or 
note  a  reason  for  not  doing  so. 

1.2 

Monitoring 

Comorbility  &  Complications 

1.2.1 

Anxiety 

Anxiety,  PL_Anxiety 

1.2.2 

School  Avio  dance 

School_Avoid,  PL_School_Avoid,  School_Phobia, 

PL  School  Phobia 

1.2.3 

Social  Isolation 

Socjso,  PL_Soc_lso, 

1.2.4 

Severe  Recurrent  Headaches 

1.2.5 

Shortness  of  breath 

Short_Breath  +  PL_Short_Breath 

1.2.6 

Exercise  Intolerance 

Activityjntol,  PL_Activity_lntol 

1.2.7 

Snoring 

Snoring,  PL_Snoring 

1.2.8 

Apnea 

Apnea,  PL_Apnea 

1.2.9 

Daytime  sleepiness 

Day_Sleepy,  PL_Day_Sleepy 

1.2.10 

Sleepiness  or  Wakefulness 

Excess_Sleep,  PL_Excess_Sleep 

1.2.11 

Abdominal  Pain 

Abdom_Pain,  PL_Abdom_Pain 

1.2.12 

Psych  Factors 

Psych_Factors,  PL_Psych_Factors 

Knowledge  items  highlighted  in  yellow  were  identified  as  having  possible  indicators  in  the  Nemours  data 
warehouse.  The  table  continues  below. 
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ID 

Domain 

Data  Elements 

Field  Name(s) 

Notes-> 

Phase  or 
diagnosis/ 
treatment 
step 

Knowledge  identified  from 
interview  and  document  content 
analysis 

Name  from  SAS  datafile 

1.2 

Monitoring 

Comorbility  &  Complications 

1.2.13 

Knee  pain 

1.2.14 

Walking  pain 

1.2.15 

Foot  pain 

1.2.16 

Hip  pain 

1.2.17 

Irregular  Menses 

1.2.18 

Primary  amenorrhea 

1.2.19 

Wheezing 

1.2.20 

Cushing’s  Syndrome 

1.2.21 

slipped  capital  femoral  epiphyses 

1.2.22 

Legg-Calve-Perthes  disease 

1.2.23 

Nocturnal  enuresis 

2.0 

Trigger 

Trigger 

2.0.1 

BMI  85th  to  94th  Percentile  -  Overweight 

2.0.2 

BMI  >  95th  Percentile  -  Obese 

3.0 

Diagnosis 

Diagnosis 

3.1 

BMI  >  95th  Percentile  -  Obese 

PN_Obese,  PI_Obese,  Obesity_DX,  PL_Obesity_DX 

4 

Investigation 

4.0 

Tests  Ordered 

4.0.1 

A1C 

Glucose 

4.0.2 

Fasting  Sugar 

BMP.  CMP 

4.0.3 

Lipid  Profile 

Lipid 

4.0.4 

Liver  Function 

Liv_Funct 

4.0.5 

When  a  primary  care  team  ordered  a  blood  test,  x-ray,  or 
other  tests,  a  follow-up  discussion  with  parents  to  provide 
those  results  should  be  documented. 

4.1 

History 

4.1.1 

Parental  Obesity 

Obes  ity_Fam  HX_DX,  PL_Obesity_FamHX_DX 

4.1.2 

Sibling  obesity 

Obes  ity_Fam  HX_DX  PL_Obesity_FamHX_DX 

Childe  Obesity  History 

Obesity_HX_DX  PL_Obesity_HX_DX 
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ID 

Domain 

Data  Elements 

Field  Name(s) 

Notes-> 

Phase  or 
diagnosis/ 
treatment 
step 

Knowledge  identified  from 
interview  and  document  content 
analysis 

Name  from  SAS  datafile 

4.1 

History 

4.1.4 

Parental  Type  2  Diabetes 

4.1.5 

Sibling  type  2  diabetes 

4.1.6 

Parental  Hypertension 

4.1.7 

Sibling  Hypertension 

4.1.8 

Child  Screen  time 

4.1.9 

Child  dietary  habits 

PI_Wt_Mngmnt,  PN_Wt_Mngmnt 

4.1.10 

Child  Activity  habits 

PI_Exercise,  PN_Exercise 

4.1.11 

Child  sleep  habits 

4.1.12 

Child  medications 

4.1.13 

Parental  Smoking 

4.1.14 

Parental  Alcohol  use 

4.1.15 

Child  alcohol  use 

4.2 

Physical 

4.2.1 

Auscultation/Pulmonary 

4.2.2 

Auscultati  on/Card  io 

4.2.3 

Auscultation/GI 

4.2.4 

Shape  of  face 

4.2.5 

Skin  color  (Neck,  axillar) 

4.2.6 

Hip  Exam 

4.2.7 

Weight 

4.2.8 

Stature 

5.0 

Treatment 

Treatment 

5.0.1 

The  primary  care  team  should  document  in  the  chart  about 
community-based  services  that  the  child  and  family  use. 

5.0.2 

Primary  care  provider  should  explain  things  in  an  easy-to- 
understand  way:  ‘In  the  last  1 2  months,  how  often  did  your 
child’s  primary  care  provider  explain  things  about  your 
child’s  health  in  a  way  that  was  easy  to  understand?” 

5.0.3 

The  primary  care  team  should  actively  involve  patient  or 
parent(s)  in  decision-making:  “When  there  was  more  than 
one  choice  for  your  child’s  treatment  or  health  care,  how 
often  did  your  child’s  primary  care  team  ask  which  choice 

5.0.4 

The  primary  care  team  should  describe  treatment  options 
adequately:  “In  the  last  12  months,  when  there  was  more 
than  one  choice  for  your  child’s  care,  did  your  child’s 
primary  care  team  give  you  enough  information  about  each 
choice?” 

5.0.5 

The  primary  care  team  should  provide  guidance  on  other 
support  services:  ‘Does  your  child’s  primary  care  team 
suggest  support  services  and  resources  outside  of  the 
practice  when  specific  needs  arise  (eg,  diagnosis  specific 
support  groups,  disability  rights  organizations)?” 

5.0.6 

The  primary  care  team  should  work  with  the  patient’s 
family  to  specifically  develop  a  management  plan  that 
includes  visit  schedules  and  communication  strategies. 

ID 

Domain 

Data  Elements 

Field  Name(s) 

Notes-> 

Phase  or 
diagnosis/ 
treatment 
step 

Knowledge  identified  from 
interview  and  document  content 
analysis 

Name  from  SAS  datafile 

5.0 

Treatment 

Treatment 

5.0.7 

The  primary  care  team  should  give  timely  referral  to 
patients:  “In  the  last  12  months,  when  your  child  needed  a 
referral  to  a  specialist,  how  often  were  you  able  to  get  the 
referral  from  your  child’s  primary  care  provider?” 

5.0.8 

The  primary  care  team  should  help  patient/parent(s) 
coordinate  care:  “In  the  last  12  months,  how  often  did  you 
get  as  much  help  as  you  wanted  with  arranging  or 
coordinating  your  child’s  care?” 

5.0.9 

The  primary  care  team  should  follow-up  with  parents  on 
visits  to  specialists:  “How  often  did  your  child’s  primary 
care  provider  or  staff  talk  with  you  about  what  happens 
during  visits  to  a  specialist  doctor?” 

Treatment 

Referral 

5.1.1 

Referall  to  Behavioral  Pediatrics 

Ref_Behav_Peds 

5.1.2 

Referall  to  Consultative  Pediatrics 

Ref_Consult_Peds 

5.1.3 

Referall  to  Nutritionist 

Ref_Nutrition 

5.1.4 

Referral  to  Weight  Management 

Ref_Wit_Mngmnt 

5.1.5 

Referral  to  Endoscopy 

Ref_Endo 

5.1.6 

Referral  to  Gastroenterology 

Ref_G  astro 

6.0 

Monitoring 

Post  Treatment  Monitoring 

6.1 

When  the  patient  is  16  y  of  age  or  older,  the  primary 
care  team  should  document  a  discussion  with  patient 
or  parent(s)  on  transitioning  to  adult  health  care 
providers. 

6.2 

At  the  point  of  transfer,  the  primary  care  team  should 
document  the  adult  care  provider  that  has  been 
identified  to  eventually  take  over  care. 

7.0 

Prevention 

Prevention  &  Control 

7.0.1 

The  primary  care  team  should  document  counseling  about 
nutrition  when  a  child’s  BMI  is  85%  percentile  for  age  and 
gender  or  note  a  reason  for  not  doing  so. 

7.0.2 

5  2  10  message,  which  reminds  families  to  eat  5  fruits  and 
vegetables,  spend  no  more  than  2  hours  on  screen  time, 
include  1  hour  of  physical  activity  or  active  play,  and 
consume  little  or  no  sugar-sweetened  beverages 

PN_521+PI_521 

8.0 

General 

Risks/Side  Effects 

8.1 

8.2 

9.0 

General 

Resources 

9.1 

Guideline  created  by  14  organizations  published  in  12.07 
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12.  CONCLUSION  -  DISCUSSION 


The  determination  of  a  performance  metric  is  only  one  step  of  the  analysis  process.  The  resulting 
performance  values  are  the  measured  outcome  of  the  observed  performance  events.  The  next  step  will 
require  development  and  evaluation  of  the  forecast  function  for  these  performance  metrics. 
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1.  Project  Background  Information 

Frequency  of  exposure  to  specific  clinical  problems  and  processes  may  mediate  military  physicians' 
maintenance  or  decay  of  clinical  knowledge  and  proficiencies.  While  deployed,  military  physicians  may 
experience  less  demand  for  certain  clinical  skills.  Upon  resuming  domestic  practice,  they  may  exhibit 
skill  decay,  reducing  health  care  quality  and  safety.  The  study  of  variables  that  influence  clinical  skill 
decay,  and  of  appropriate  interventions,  is  certainly  valuable.  We  propose  a  program  of  sequential 
research  that  leverages  the  unique  combined  resources  of  Nemours,  one  of  the  nation's  largest 
pediatric  health  care  systems,  and  the  Lockheed  Martin  Corporation,  with  its  extensive  expertise  in 
the  measurement  and  amelioration  of  skill  decay  among  pilots  and  other  military  personnel. 

2.  Summary  of  Achievements  -  Year  2:  Validate  a  multivariate  model  of  physicians1  skill  decay  as  a 

function  of  frequency  of  exposure  to  the  targeted  clinical  processes 

The  Lockheed  Martin  team  accomplished  the  following  tasks  during  this  phase: 

•  At  least  one  member  of  the  Lockheed  Martin  team  participated  in  scheduled  conference  calls 
hosted  my  Nemours  for  the  entire  research  team  and  Face-to-Face  Workshop  meetings. 

•  At  least  one  member  of  the  Lockheed  Martin  team  participated  in  scheduled  conference  calls 
of  assigned  working  groups. 

•  The  team  review,  edit  and  discussed  documents  distributed  for  commentary.  This  is 
documented  in  meeting  minutes  located  on  the  Lockheed  Martin-hosted  Project  Collaboration 
SharePoint  at:  https://gtl- 

dom.external.lmco.com/sites/TLS  Prof  Proiect/Shared%20Documents/Forms/Allltems.aspx . 

•  Served  as  a  resource  to  the  remainder  of  the  research  team  in  terms  of  familiarizing  the  team 
with  pertinent  theoretical  and  empirical  literature  about  decay  in  complex  cognitive  skills  that 
can  be  applied  to  understanding  of  clinical  decision  making  in  health  care. 

•  Consulted  with  the  research  team  in  evaluation  of  the  feasibility,  acceptability  and 
psychometric  validity  of  measurement  of  the  predictor  and  criterion  variables  that  comprise 
the  multivariate  model(s)  to  be  tested. 

•  Contributed  to  the  design  and  execution  of  process  improvement  activities  to  address  any 
issues  identified  above. 

•  Collaborated  with  the  project  statistician  to  develop  a  comprehensive  statistical  analysis  plan 
that  will  enable  sound  evaluation  of  each  model  to  be  tested  as  well  as  comparison  of  any 
alternative  explanatory  models. 

•  Continued  data  collection  to  obtain  the  data  specified  by  any/all  models  to  be  tested, 
definitely  including  cross-sectional  models  and  possibly  including  prospective  models. 

•  Collaborated  with  the  project  statistician  to  implement  and  interpret  the  results  of  model 
testing  and  to  use  this  analysis  to  guide  planning  of  interventions  to  be  tested  in  Year  3. 

•  Worked  with  the  research  team  to  document  initial  considerations  of  alternative  intervention 
components  including  electronic  health  record  decision  supports;  use  of  avatar-based  or  other 
case  simulation  educational  experiences. 

•  Supported  the  US  Army  customer's  In  Progress  Review  (IPR)  that  took  place  in  ll-Aug-2015  in 
Ft.  Detrick,  MD. 


3.  Project  Meetings 

The  following  teleconference  meetings  were  held  over  the  course  of  Year  2  in  hosted  by  the  Modeling 
and  Analysis  Working  Group  (Note:  SharePoint  link  to  meeting  agenda  is  included  in  the  table  below. 
Minutes  can  be  found  in  the  same  SharePoint  folder): 


MEETING  SUBJECT 

DATE 

APPROX. 

DURATION 

1.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2014  0925  Agenda  DOD- 

25-Se  p-2014 

1.0  Hour 
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MEETING  SUBJECT 

DATE 

APPROX. 

DURATION 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

2.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2014  1009  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

09-0ct-2014 

1.0  Hour 

3.  Nemours/LM  Face-to-Face  Workshop  at 
Lockheed  Martin  MST  Facility,  Orlando  FL 

1.5  Days 

Hosted  by: 

•  Lorie  Ingraham 

•  Jim  Crutchfield 

23-24  Oct-2014 

4.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2014  1113  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

ll-Nov-2014 

1.0  Hour 

5.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2014  1120  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

20-Nov-2014 

1.0  Hour 

6.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  -  Special  Interim  Meeting 
(TELECON) 

2014  1211  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

12-Dec-2014 

1.0  Hour 

7.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0108  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

8-Jan-2015 

1.0  Hour 

8.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0122  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

22-Jan-2015 

1.0  Hour 

9.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0212  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

12-Feb-2015 

1.0  Hour 

10.  DOD-Model  Building  and  Analysis  Work 

1.0  Hour 
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MEETING  SUBJECT 

DATE 

APPROX. 

DURATION 

Group  Meeting  (TELECON) 

2015  0226  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

•  Jim  Crutchfield 

26-Feb-2015 

11.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0312  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

12-Mar-2015 

1.0  Hour 

12.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0326  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

26-Mar-2015 

1.0  Hour 

13.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0409  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

9-Apr-2015 

1.0  Hour 

14.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0423  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

23-Apr-2015 

1.0  Hour 

15.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0528  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

28-May-2015 

1.0  Hour 

16.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0611  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

ll-Jun-2015 

1.0  Hour 

17.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0618  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

18-Jun-2015 

1.0  Hour 

18.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0709  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

9-Jul-2015 

1.0  Hour 

19.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0723  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

23-Jul-2015 

1.0  Hour 
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MEETING  SUBJECT 

DATE 

APPROX. 

DURATION 

Jim  Crutchfield 

20.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0813  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

13-Aug-2015 

1.0  Hour 

21.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0827  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

27-Aug-2015 

1.0  Hour 

22.  DOD-Model  Building  and  Analysis  Work 
Group  Meeting  (TELECON) 

2015  0910  Agenda  DOD- 

Model  Building  and  Analysis  WG  Meeting.docx 

Hosted  by: 

Jim  Crutchfield 

10-Se  p-2015 

1.0  Hour 

4.  Project  Deliverables 

The  following  documents  were  delivered  during  Year  2: 

1.  Documented  participation  of  at  least  one  Lockheed  Martin  team  member  at  scheduled 
meetings  or  conference  calls  by  the  entire  research  team  or  working  groups  as  demonstrated 
by  meeting  minutes. 

2.  Delivered  in  March  2015  a  formal  written  proposal  of  the  measurement  protocol  for  collection 
of  the  data  needed  to  test  and  compare  the  validity  of  one  or  more  alternative  quantitative 
models  of  variables  associated  with  skill  decay  of  pediatric  health  care  providers. 

3.  Participated  in  editing  and  conceptual/technical  refinement  of  statistical  analysis  plans  for 
evaluation  of  the  proposed  quantitative  explanatory  model(s)  as  evidenced  by  tracked 
changes  and  comments  inserted  in  successive  versions  of  such  documents. 

4.  Developed  a  written  data  management  plan  that  ensures  the  integrity  of  collected  data  and 
that  provides  procedures  for  data  verification,  quality  control,  recognition  of  outliers  and  out 
of  range  values  and  for  treatment  of  missing  data. 

5.  Participated  in  conference  calls  or  video  conferences  with  the  project  statistician  during  the 
3rd  and  4th  quarters  of  Year  2  to  achieve  appropriate  interpretation  of  the  multivariate 
analyses  of  the  quantitative  model(s)  being  tested  (documented  in  analysis  write-ups). 

6.  Participated  with  other  members  of  the  research  team  in  conceiving,  drafting,  editing  and 
submitting  journal  articles,  abstracts,  posters  and  conference  presentations  reporting  the 
results  of  the  model  building  and  testing  activities.  The  Lockheed  Martin  team  particularly 
contributed  to  the  completion  of  two  (2)  accepted  publications  in  Year  2: 

•  Military  Operational  Research  Society  Symposium  (2015  MORSS)  Paper;  Titled: 
"Validation  of  cognitive  performance  measurements  to  map  physician  declarative 
knowledge  in  practice" 

•  Interservice/Industry  Training,  Simulation  and  Education  Conference  (l/ITSEC  2015); 
Title:  "Structured  Development  of  Interventions  to  Improve  Physician  Knowledge 
Retention" 

7.  Participated  with  other  members  of  the  research  team  in  decision  making  about  the  design  of 
the  Year  3  intervention  trial  guided  by  the  quantitative  model  testing  results. 

YEAR  2  End  Project  Report  files  (3)  -  Deliverables: 

[1]  Healthcare  Skill  Maintenance  Model  Performance  Metric  Construction 
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(Ms  Word  Doc) 

Filename:  "HealthcareSkillsetPerformanceMetrics_FormalReport.docx" 

[2]  Research  Data  Management  Plan  (MS  Word  Doc) 

Filename:  "ResearchDataManagementPlan.docx"  including  two  (2)  spreadsheets 
DataDictionary.xIsx  and  DKM_v8.xlsx  in  zip. allow  file  (please  remove  ".allow"  to  open) 

[3]  This  file:  Army  Med  Nemours  LM  YEAR  2  Project  Report  -  Sep  2015.doc 


5.  External  Dependencies 

The  LM  team  requires  from  Nemours  the  following: 

•  Electronic  Medical  Record  data  extractions  approved  for  release  to  the  research  team 

•  Access  to  a  SAS  Analytical  Computing  environment 

•  Intervention  content  for  prototype  decision  support  aids  and  simulations. 

6.  Notes 

It  has  been  a  pleasure  and  a  privilege  to  be  working  with  the  Nemours  Team  in  Year  2.  We  look 
forward  to  continued  success  in  Year  3. 
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Abstract 


The  Electronic  Health  Record  (EHR)  has  the  potential  to  provide  insight  into  possible  decay  in 
health  care  providers’  (HCP)  clinical  knowledge  or  cognitive  performance.  Analyses  of  the  contributions 
of  variables  such  as  frequency  of  exposure  to  targeted  clinical  problems  could  inform  the  development 
and  testing  of  appropriate  interventions  to  mitigate  these  threats  to  quality  and  safety  of  care. 

De-identified,  aggregated  study  data  were  obtained  for  2013.  Nine  targeted  clinical  problems 
(TCP)  were  selected  for  further  study.  Task  analysis  interviews  of  subspecialty  physicians  defined 
optimal  management  of  each  TCP  and  specified  quality  of  care  metrics  that  could  be  extracted  from  the 
EHR.  The  A-t  statistic,  days  since  the  provider’s  prior  encounter  with  a  given  TCP,  quantified  frequency 
of  exposure. 

Frequency  of  patient  encounters  ranged  from  1 ,566  to  220,774  across  conditions.  Mean  A-t 
ranged  from  1 .72  days,  to  30.79  days.  Maximum  A-t  ranged  from  285  to  497  days.  The  distribution  of 
A-t  for  the  TCPs  generally  fit  a  Gamma  distribution  (P  <  0.001 ),  indicating  that  A-t  conforms  to  a 
Poisson  distribution.  A  quality  of  care  metric  derived  for  each  TCP  declined  progressively  with 
increasing  A-t,  affirming  that  knowledge  decay  was  detectable  from  EHR  data. 

This  project  demonstrates  the  utility  of  the  EHR  as  a  research  tool  in  studies  of  health  care 
delivery  in  association  with  frequency  of  exposure  of  HCPs  to  TCPs.  Subsequent  steps  in  our  research 
include  multivariate  modeling  of  clinical  knowledge  decay  and  randomized  trials  of  pertinent  preventive 
interventions. 
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INTRODUCTION 


Along  with  the  steady  increase  in  the  adoption  of  electronic  health  records  (EHR)  over  the  past 
decade,  EHR  functionality  has  shifted  from  basic  systems  to  greater  functionality  featuring 
computerized  provider  order  entry,  result  management  and  decision  support.  The  government’s 
encouragement  of  “Meaningful  Use”  of  EHR  systems  backed  by  financial  incentives,  the  growth  of  the 
Learning  Health  System  movement,  and  the  projected  evolution  of  the  health  care  marketplace  from  a 
quantity-driven  to  a  value-driven  model  are  among  many  influences  that  will  revolutionize  the  current 
U.S.  Healthcare  System  (1-3).  As  EHR  becomes  increasingly  commonplace  and  institutions  and  health 
care  providers  become  increasingly  facile  in  using  and  customizing  tools  to  document  clinical 
encounters,  there  is  a  growing  new  capacity  to  store  discrete  clinical  data  elements  that  can 
subsequently  be  mined  for  clinical  information.  Specifically,  the  systematic  use  of  EHR  data  will  be 
used  for  continuous  improvement  and  research  purposes  to  enhance  quality,  safety,  outcomes,  cost 
and  efficiency  of  care,  while  reducing  waste,  errors  and  duplication.  In  this  paper,  we  describe  our 
ongoing  efforts  to  utilize  routinely  collected  EHR  data  as  a  window  into  clinicians’  decision  making 
regarding  the  management  of  nine  common  pediatric  conditions. 

The  pace  of  advances  in  medical  knowledge  continues  to  accelerate  and  health  care  providers  are 
challenged  with  assimilating  new  knowledge  while  retaining  what  has  already  been  learned.  (4) 
Performance  decay  is  inevitable  and  yet  physicians  cannot  accurately  assess  their  own  knowledge 
proficiency.  (5)  In  the  health  professions,  quality  of  care  (QOC)  is  most  critical  from  both  a  human  and  a 
business  perspective.  The  cumulative  costs  associated  with  low  quality,  waste,  delay,  and  redundancy 
in  care  are  great.  Similar  issues  are  encountered  where  high  reliability  is  required  including  the  military, 
energy  and  transportation  sectors.  (3,6) 

In  an  era  of  rising  healthcare  costs,  we  are  investigating  a  cost  effective  approach  to  QOC 
performance  sustainment  by  using  forecast  models  to  provide  an  individualized  guide  to  the  application 
of  interventions  that  will  counter  the  effects  of  knowledge  decay.  Further,  clinical  declarative  knowledge 
can  be  inferred  by  the  demonstration  of  a  caregiver’s  ability  to  recall  and  employ  relevant  knowledge  for 
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the  diagnosis  and  proper  care  of  patients  at  the  point  of  care,  at  the  time  that  knowledge  is  required. 
Decay  in  relevant  clinical  cognitive  performance  is  evident  if  EHR  derived  measures  of  clinical 
declarative  knowledge  decline  over  time,  for  instance  with  longer  duration  since  the  prior  clinical 
exposure  to  the  clinical  problem  of  interest. 

This  paper  describes  a  process  that  determines  where  the  EHR  archive  provides  sufficient  data  for 
knowledge  retention  forecast  modeling  and  the  selection  of  a  limited  set  of  key  clinical  issues  for  further 
investigation.  In  medical  services  and  other  High  Reliability  Organizations  (HRO),  QOC  improvement 
typically  yields  benefits  in  effectiveness.  (6)  We  believe  this  approach  could  further  improve  that 
equation  by  targeting  for  intervention  specific  people,  circumstances  or  elements  of  the  care  process 
that  may  yield  the  greatest  benefit  in  terms  of  quality,  safety,  outcomes  and  costs  of  care. 

Therefore,  the  initial  objective  of  the  work  reported  in  this  paper  was  to  find  elements  of  the  care 
processes  for  certain  targeted  clinical  problems  (TCP)  that  demonstrate  risk  for  QOC  performance 
decay  that  can  be  identified  from  EHR  data.  We  sought  to  identify  health  care  service  issues  where 
existing  data  enable  a  forecast  model  of  declarative  knowledge  decay.  In  this  context,  we  define 
declarative  knowledge  as  the  verbal,  aural  and  visual  information  that  is  used  by  an  HCP  when  relevant 
and  necessary  for  the  proper  care  of  patients.  The  paper  then  describes  the  distributions  of  duration  of 
time  since  the  prior  exposure  to  a  given  targeted  clinical  problem  (the  A -t  statistic)  and  affirms  that  there 
was  sufficient  variability  in  this  statistic  to  support  additional  analyses.  The  paper  concludes  with 
analyses  of  associations  between  quality  of  care  metrics  for  each  TCP  with  increasing  values  of  the  A -t 
statistic.  Initial  specification  of  TCPs  for  further  study,  and  initial  validation  of  the  associations  between 
clinical  knowledge  decay  and  the  A -t  statistic  are  prerequisites  to  the  subsequent  phases  of  this 
research  program. 

The  next  step  in  this  ongoing  program  of  research  consists  of  evaluating  multivariate  predictive 
models  of  decay  in  HCP’s  clinical  declarative  knowledge  and  cognitive  performance,  including 
frequency  of  exposure  to  the  TCPs  as  a  key  predictor  variable.  That  work  will  be  followed  by  trials  of 
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pertinent  interventions  (e.g.  job  performance  aids,  education,  computerized  decision  supports,  case 
simulation  experiences,  etc.)  designed  to  prevent  or  minimize  the  influences  of  low  frequency  of 
exposure  and  other  identified  predictor  variables  on  clinical  knowledge  decay. 

Method 

Setting 

The  activities  described  in  this  paper  took  place  throughout  Nemours  Children’s  Health  System, 
which  owns  and  operates  two  free-standing  children’s  hospitals  (Alfred  I.  duPont  Hospital  for  Children 
in  Wilmington,  Delaware  and  Nemours  Children’s  Hospital  in  Orlando,  Florida),  as  well  as  pediatric 
subspecialty  clinics  and  primary  care  networks  throughout  the  Delaware  Valley  and  Florida.  More  than 
700  doctoral  level  health  care  providers  and  198  advanced  practice  registered  nurses  (APRN)  are 
employed  by  the  organization.  All  of  the  operating  entities  have  utilized  the  same  EHR  (EPIC, 

Madison,  Wisconsin)  since  the  late  1990’s  and  the  organization  supports  a  single  EHR  Data 
Warehouse.  The  system  provides  more  than  300,000  health  care  encounters  annually  to  infants, 
children  and  adolescents  in  these  locations.  The  organization  has  received  awards  from  the  Healthcare 
Information  Management  System  Society  (HIMSS)  for  excellence  in  its  commitment  to  health 
information  technology  and  has  consistently  earned  high  marks  for  its  performance  relative  to  the 
HITECH  Meaningful  Use  standards  since  those  were  implemented  in  2011.  The  project  described  in 
this  paper  also  involved  a  collaborative  relationship  with  staff  of  the  Mission  Systems  and  Training 
division  of  the  Lockheed-Martin  Corporation,  which  operates  a  state  of  the  art  human  performance 
engineering  and  simulation  center  in  Orlando,  Florida.  That  group  brought  special  expertise  to  the 
project  in  terms  of  completion  of  detailed  task  analyses  of  optimal  performance  of  complex  skills  via 
structured  interviews  of  subject  matter  experts,  identification  of  worker  and  workplace  characteristics 
that  might  influence  clinical  skill  decay/maintenance,  analysis  of  quantitative  modeling  of  skill 
decay/maintenance  and  matching  of  intervention  strategies  to  characteristics  of  specific  clinical  skills. 
Procedure 
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The  team’s  efforts  to  determine  TCPs  for  further  study  began  with  examination  of  common 
diagnoses  recorded  in  administrative  data.  First,  the  top  100  most  frequently  encountered  outpatient 
diagnoses  were  identified.  Conditions  with  multiple  related  diagnoses  were  aggregated.  For  example, 
asthma  exacerbation  with  and  without  status  asthmaticus,  as  well  as  mild,  moderate,  and  severe 
asthma  were  classified  together.  Second,  the  Nemours  Data  Warehouse  was  then  queried  to  extract 
the  data  necessary  to  further  evaluate  each  of  these  clinical  problems  using  the  following  criteria: 

•  Frequency  with  which  the  specific  clinical  problem  was  entered  at  a  documented  outpatient 
clinical  encounter.  This  was  analyzed  to  ensure  that  adequate  multivariate  analyses  could  be 
performed  as  rarely  encountered  clinical  problems  may  not  provide  enough  data  to  allow  for  this 
assessment. 

•  Number  of  Divisions  across  the  entire  Nemours  Health  System  seeing  the  clinical  problem  and 
the  varied  types  of  health  care  providers  responsible  for  the  above  visits,  for  example, 
subspecialist  physicians,  primary  care  physicians,  advanced  practice  nurses,  and  physician 
assistants.  This  was  queried  to  ensure  variability  in  expertise  and  frequency  of  exposure. 

•  Variability  among  providers  in  terms  of  frequency  of  exposure  to  the  clinical  problem,  again  to 
facilitate  robust  and  productive  multivariate  analyses. 

•  Number  of  unique  patients  presenting  with  the  clinical  problem  per  year  and  number  of 
condition-related  encounters/patient/year.  This  was  also  assessed  to  determine  variability  in 
frequency  of  exposure. 

•  Availability  of  adequate  EHR  data  to  permit  quantification  of  frequency  of  exposure. 

•  The  presence  of  controversy  regarding  optimal  clinical  management  of  the  problem  was  not  an 
exclusion  factor  and  was  thought  to  contribute  to  variability  among  providers  in  terms  of  what 
management  steps  are  taken. 

The  research  team  iteratively  reviewed  the  list  of  candidate  clinical  problems  to  maximize  exposure 
to  the  above  factors  which  resulted  in  a  list  of  12  candidate  conditions.  Third,  clinical  leaders  from  the 
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pertinent  clinical  divisions/services  were  enlisted  to  serve  as  Subject  Matter  Experts  (SMEs)  to  assist  in 
further  selection  of  targeted  clinical  problems  and  to  assist  the  team  with  defining  optimal  clinical 
management  of  the  targeted  clinical  problem  under  consideration.  The  latter  was  achieved  through 
structured  task  analysis  interviews  of  SMEs,  often  supplemented  with  published  evidence-based 
treatment  guidelines,  systematic  reviews,  and  organizational  policies  and  position  statements. 

From  the  initial  list  of  12  candidate  conditions,  the  research  team  selected  9  targeted  clinical 
problems  for  further  study  based  on  the  team’s  capacity  and  the  identification  of  physician  champions 
for  the  respective  TCPs.The  selected  TCPs  included  gastroesophageal  reflux  disease  (GERD), 
headache,  concussion,  idiopathic  scoliosis,  encopresis/constipation,  influenza  vaccination, 
supracondylar  fracture,  Type  1  diabetes  and  obesity.  The  team  developed  a  A-t  statistic,  the  number  of 
days  since  the  prior  encounter  with  the  index  clinical  problem,  as  its  measure  of  frequency  of  exposure. 
In  addition  to  the  A-t  statistic,  a  number  of  measures  of  worker  characteristics  (provider  age, 
race/ethnicity,  years  of  experience,  nation  of  origin,  first  language,  board  certifications,  EHR 
proficiency)  and  workplace  characteristics  (#  patients  scheduled  on  date  of  visit,  day  of  week  of  visit, 
time  of  visit,  clinician’s  on-call  responsibility)  have  been  obtained  as  additional  moderators  and 
mediators. 

Finally,  indices  reflecting  provider  decision  making  about  the  clinical  management  of  the  index 
problems  were  extracted  from  the  Data  Warehouse  for  eventual  entry  into  quantitative  models.  This 

was  achieved  by  first  conducting  separate  Principal  Components  Analyses  for  each  TCP,  including  in 
those  analyses  the  various  EHR  data  elements  specified  in  the  task  analysis  interviews  of  the  SMEs.  In 

order  to  ensure  that  the  analysis  was  examining  knowledge  decay  rather  than  HCP  nonadherence  with 
optimal  care  practices,  the  knowledge  metric  scores  that  entered  these  analyses  consisted  of  the 
maximum  score  obtained  at  any  given  value  of  the  A -t  statistic.  (Results  were  similar  when  all  scores 
were  included  in  the  analyses).  The  measurement  factor  that  was  selected  for  each  TCP  was  the  factor 
that  emerged  as  having  the  highest  eigenvalue  for  that  TCP  (range  XX. XX  to  YY.YY).  For  each  such 

measurement  factor,  EHR  documentation  of  each  encounter  for  each  HCP  with  the  TCP  of  interest  was 
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scored  in  terms  of  the  presence/absence  of  each  of  these  data  elements  and  the  total  score  was 
treated  as  a  composite  quality  metric  for  that  encounter.  Then,  the  distributions  of  scores  on  these 

quality  metrics  for  each  TCP  was  plotted  as  a  function  of  the  spectrum  of  values  of  the  A -t  statistic 
derived  for  those  same  encounters. 

RESULTS 

The  nine  clinical  problems  we  selected  were  evaluated  across  the  Nemours  Children’s  Health 
System,  using  EHR  data  from  2013.  Table  1  illustrates  the  successful  implementation  of  the  criteria 
employed  to  target  clinical  problems  for  further  study.  Nine  diverse  clinical  problems  were  selected, 
each  of  which  were  seen  by  care  providers  representing  a  large  number  of  clinical  sub-units  (i.e., 
departments,  divisions,  and  satellite  clinic  locations)  (Mean  =  61.8  sub-units;  range  28-226  across 
clinical  problems).  A  total  of  1,735  different  health  care  providers  offered  care  to  these  patients  (MD: 
298,804  encounters;  82.3%;  DO:  29,855  encounters;  8.2%;  and  APRN:  34,341  encounters;  9.4%).  The 
nine  targeted  clinical  problems  were  all  seen  commonly  (Mean  number  of  distinct  patients  per  targeted 
clinical  problem  =  16,412.9;  mean  number  of  encounters  per  targeted  clinical  problem  =  40,333.3). 

Mean  A-t  ranged  from  1 .72  days  for  obesity,  to  30.71  days  for  influenza  vaccination.  In  most 
cases,  A-t  represents  the  time,  in  days,  between  the  last  encounter  with  a  patient  with  the  specified 
clinical  problem.  For  the  influenza  vaccination,  A-fwas  computed  against  the  latest,  previous 
administration  of  vaccination,  since  in  most  cases,  the  patient  is  not  present  solely  for  that  issue.  As 
would  be  expected,  based  on  the  lower  number  of  cases,  encopresis/constipation  appeared  with  the 
next  highest  mean  A-t  of  21 .86  days.  Maximum  A-t  ranged  from  285  days  for  obesity  to  497  days  for 
headaches.  Obesity,  encopresis/constipation  and  GERD  demonstrated  a  maximum  A-t  of  somewhat 
less  than  a  year,  but  the  rest  exceeded  400  days.  As  seen  in  Figures  1 , 2  and  3,  the  distribution  of  A-t 
fit  a  Gamma  distribution  (P  <  0.001),  indicating  that,  except  for  influenza  vaccination,  the  intervals 
between  successive  patient  encounters  for  these  clinical  problems  follow  a  Poisson  distribution.  The 
distribution  for  influenza  vaccination  differed  very  slightly  due  to  seasonality  of  administration,  but  still 
generally  fit  the  distribution. 
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Additional  exploratory  analyses  were  performed  to  determine  whether  the  derived  measures  of 


HCP’s  clinical  knowledge  varied  systematically  at  different  values  of  the  A -t  statistic.  Figure  4  portrays  a 


representative  illustration  of  these  analyses  in  displaying  the  maximum  score  obtained  on  the 


Concussion  knowledge  metric  as  a  function  of  increasing  values  of  the  A -t  statistic.  As  shown  in  Figure 


4,  the  clinical  knowledge  score  declined  in  a  curvilinear  manner  with  respect  to  longer  duration  since 


the  prior  exposure  to  Concussion  as  the  presenting  clinical  problem.  Table  2  (or  Figure  5)  summarizes 

the  corresponding  data  for  the  other  8  TCPs,  each  of  which  also  demonstrated  statistically  significant 


decline  in  the  clinical  knowledge  metric  with  increasing  values  of  the  A -t  statistic 


DISCUSSION 

EHR  technology  increasingly  penetrates  into  the  routine  operations  of  health  care  entities,  (4,  7) 
and  these  entities  will  increasingly  compete  in  a  value-driven  health  care  marketplace  as  defined  by  the 
U.S.  Department  of  Health  and  Human  Services.  Health  care  organizations  that  can  utilize  their 
accumulated  EHR  data  to  improve  the  quality  and  safety  of  care  they  deliver,  while  reducing  waste, 
delay  and  redundancy  will  be  best  positioned  to  thrive  in  that  environment.  The  work  reported  here 
describes  the  first  step  in  an  initiative  that  seeks  to  determine  if  frequency  of  exposure  to  targeted 
clinical  problems  is  a  risk  factor  for  decay  in  HCP’s  clinical  declarative  knowledge  and  if  timely,  cost- 
efficient  interventions  can  favorably  impact  the  expression  of  that  risk.  This  paper  describes  the  first 
step  in  a  planned  program  of  research  to  determine  if  frequency  of  exposure  to  targeted  clinical 
problems  can  be  reliably  quantified  based  on  historic  EHR  data  and  if  that  measure  can  be  shown  to  be 
associated  significantly  with  unwanted  variability  in  specific  care  practices.  The  research  team 
implemented  a  systematic  process  to  select  targeted  clinical  problems  using  a  combination  of  empirical 
criteria  and  careful  characterization  of  the  opinions  of  subject  matter  experts.  The  collection  and 
analysis  of  substantial  EHR  data  permitted  derivation  of  an  index  of  the  duration  of  time  in  days  (the  A-t 
statistic)  since  a  given  HCP’s  prior  exposure  to  a  given  targeted  clinical  problem.  After  consideration  of 
numerous  candidate  problems,  the  nine  targeted  clinical  problems  that  were  selected  for  detailed 
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analysis  included  obesity,  headache,  idiopathic  scoliosis,  encopresis/constipation,  gastroesophageal 
reflux  disease,  concussion,  influenza  vaccination,  type  1  diabetes  and  supracondylar  fractures.  While 
reflecting  diversity  of  pathology,  assessment,  management  and  disease  course,  these  conditions 
shared  certain  features,  including:  relatively  high  frequency  of  presentation  in  our  health  care  system; 
multiple  different  types  of  HCPs  involved  in  the  delivery  of  care;  variability  in  frequency  of  exposure  to 
the  problem  across  HCPs;  substantial  consensus  among  subject  matter  experts  regarding  elements  of 
appropriate  management  of  each  problem;  variability  between  and  within  providers  regarding  the 
degree  to  which  actual  care  practices  conform  to  evidence-based  care;  EHR  data  of  sufficient  quality 
and  quantity  to  permit  the  planned  analyses;  and  adequate  commitment  from  affected  clinical  divisions 
and  services  to  collaborate  on  this  project. 

Distributions  of  the  A-t  statistic  varied  among  the  targeted  clinical  problems.  These  distributions 
approximated  a  Gamma  distribution  (P  <  0.001)  for  each  targeted  clinical  problem.  However,  the  A-t 
distribution  for  influenza  vaccination  differed  somewhat  in  shape  from  those  for  the  other  targeted 
clinical  problems  due  to  the  seasonality  of  opportunities  for  such  encounters. 

A  clinical  knowledge  metric  specific  to  each  TCP  was  derived  from  factor  analyses  of  the  various 


data  elements  suggested  by  the  SMEs  as  indicators  of  optimal  clinical  assessment  and  decision 


making  for  each  TCP.  Exploratory  analyses  showed  that  maximum  scores  obtained  on  this  knowledge 

metric  declined  significantly  with  increasing  values  of  the  A-t  statistic  for  all  9  of  the  TCPs.  These 

results  suggest  that: 

•  Clinical  knowledge  decay  occurs  with  lower  frequency  of  exposure  to  specific  clinical  problems 

and  that  data  can  be  extracted  from  an  electronic  health  record  to  identify  both  suboptimal 

levels  of  clinical  knowledge:  and 

• 

It  may  be  possible  to  identify  specific  HCPs  who  are  at  higher  risk  of  subsequent  decay  in 

clinical  knowledge  due  to  longer  duration  of  time  since  the  prior  exposure  to  a  given  clinical 

problem. 
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This  could  facilitate  both  the  development  and  validation  of  a  wide  range  of  preventive  and  remedial 
interventions  that  could  ultimately  promote  higher  quality,  safety  and  efficiency  of  care. 

Use  of  the  EHR  to  characterize  frequency  of  exposure  of  HCPs  to  specific  health  care  problems  will 
allow  health  care  organizations  to  be  better  prepared  to  identify  intervention  loci  that  can  lead  to 
improved  quality  and  safety  of  care  in  a  cost  efficient  manner.  Targeting  interventions  to  individual 
clinicians  at  risk  for  clinical  knowledge  or  cognitive  performance  decay  through  predictive  modeling 
would  be  more  efficient  and  likely  more  productive  than  a  broad  training  program  to  an  entire  medical 
staff.  Identification  of  HCPs  at  elevated  risk  of  clinical  knowledge  decay  could  prompt  targeted  provision 
of  preventive  education  for  example  job  performance  aids  such  as  procedural  checklists,  computerized 
decision  supports  at  the  moment  of  care,  or  simulation-based  training. 

Although  the  impact  of  time  between  successive  exposures  to  a  clinical  condition  has  been  the 
focus  of  this  investigation,  modeling  the  decay  of  clinical  knowledge  and  cognitive  performance  will  be 
an  iterative  process  that  will  require  consideration  of  co-factors  such  as  clinician  characteristics  (e.g. 
age,  education,  and  baseline  knowledge),  practice  characteristics  (e.g.  specialty  and  workflow),  and 
system  characteristics  (e.g.  availability  of  resources).  Subsequent  steps  in  our  research  include  building 
a  multivariate  predictive  model  of  clinical  skill  decay  and  testing  interventions  focused  on  preventing 
decay  of  clinical  knowledge. 

The  primary  limitations  of  this  work  are  the  recognition  that  the  quality  of  care  that  is  actually 
delivered  and  the  EHR  documentation  of  that  care  may  diverge  and  the  difficulty  inherent  in  attempting 
to  distinguish  between  decay  in  clinical  knowledge  versus  simple  noncompliance  by  HCPs.  We  address 
these  two  points  sequentially  below.  Also,  it  is  important  to  recognize  that  this  paper  (See  Figure  4) 
demonstrated  clinical  knowledge  decay  by  analyzing  aggregated  cross-sectional  data  that  revealed 
increasing  decrements  in  clinical  performance  in  association  with  time  since  the  HCP’s  prior  exposure 
(A-t)  to  that  TCP.  This  is  different  from  a  prospective  demonstration  that  the  magnitude  of  decay  in 
clinical  knowledge  of  individual  HCPs  is  directly  proportional  to  the  amount  time  since  that  HCP’s  prior 
exposure  to  that  TCP.  A  study  of  that  type  would  require  many  repeated  observations  of  the  pertinent 
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quality  of  care  metrics  and  the  A- 1  statistic  for  successive  clinical  encounters  for  the  same  HCPs  over 
time. 

In  certain  instances,  such  as  whether  a  specific  medication  or  laboratory  test  is  ordered,  EHR 
documentation  is  very  likely  to  be  an  extremely  valid  indicator  of  the  actual  events  transpiring  during  a 
care  encounter.  However,  an  HCP’s  failure  to  document  the  rationale  for  departing  from  accepted  care 
practices  in  a  specific  instance  could  be  interpreted  as  suboptimal  care  when  it  was  actually  appropriate 
clinically.  As  EHR  use  becomes  increasingly  typical  in  health  care,  as  HCPs  become  increasingly  facile 
with  its  use,  and  as  natural  language  processing  technology  advances  the  capacity  to  analyze  narrative 
text  contained  in  progress  notes,  the  utility  of  the  EHR  as  a  tool  for  analyzing  and  facilitating  quality, 
safety  and  efficiency  of  care  will  likely  grow  accordingly.  (8)  Organizations  that  are  well-positioned  to 
capitalize  on  the  opportunities  afforded  by  those  developments  will  be  able  to  compete  more  effectively 
in  the  coming  value-driven  health  care  economy. 

Decrements  in  HCP’s  clinical  performance  as  evidenced  by  lower  scores  on  the  metrics  that  were 
collected  as  indicators  for  this  study  could  presumably  reflect  decay  of  clinical  knowledge  as  well  as 
HCP  noncompliance  with  currently  accepted  standards  of  care.  We  attempted  to  circumvent  this 
interpretive  complication  by  focusing  our  analyses  on  the  maximum  score  on  our  knowledge  metric  for 
each  TCP  at  each  value  of  the  A-t  statistic.  It  is  implausible  that  compliance  with  accepted  standard 

clinical  practice  related  to  each  TCP  should  deteriorate  systematically  in  association  with  increasing  A-t 

but  quite  plausible  that  clinical  knowledge  would  do  so. 

Subsequent  steps  in  our  research  include  building  a  multivariate  model  to  enable  prediction  of 
clinical  skill  decay/maintenance  and  to  test  a  range  of  interventions  focused  on  preventing  decay  of 
clinical  knowledge  and  skills.  Data  collection  for  the  model-testing  phase  includes  extraction  of 
extensive  2013  EHR  data  relative  to  each  targeted  clinical  problem  as  well  as  available  data  on  worker 
characteristics  (e.g.,  HCP  demographics;  training  and  certifications;  years  of  clinical  experience;  EHR 
proficiency)  and  workplace  characteristics  (e.g.,  density  of  clinic  schedule;  temporal  distribution  of 
encounters  early  or  late  in  the  day;  on-call  previous  night  versus  not).  Model  building  and  analysis  of 
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the  2013  data  will  be  followed  by  prospective  evaluation  of  the  model  during  the  final  phase  of  this 
project  when  a  variety  of  job  aids,  educational,  decision-support  and  simulation  interventions  will  be 
tested  for  feasibility,  acceptability  and  efficacy.  Collectively,  our  work  will  be  used  to  identify  and  target 
specific  areas  where  quality  of  care  (QOC)  improvement  efforts  align  with  “Meaningful  Use”  EHR 
systems  through  a  High  Reliability  Organization  and  continue  to  move  towards  a  value-driven  U.S. 
Healthcare  EHR  system  model  that  can  have  the  greatest  payoff  and  offer  cost  effective  solutions. 
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Table  1.  2013  descriptive  statistics  for  each  of  the  9  targeted  clinical  problems. 


Condition 

Obesity 

Influenza 

Vaccination 

GERD 

Encopresis 

Constipation 

Headache 

Concussion 

Supracondylar 

Fracture 

Idiopathic 

Scoliosis 

Type  1 
Diabetes 
Mellitus 

Encounters 

89753 

220357 

25453 

1633 

9686 

2977 

2285 

10025 

5982 

Mean  delta-t 
(days) 

1.72 

30.71 

3.66 

21.86 

7.11 

13.21 

11.58 

6.78 

4.81 

Max  delta-t 
(days) 

285 

470 

336 

349 

497 

470 

475 

409 

437 

#  Distinct 
Patients 

34737 

87114 

9010 

880 

5564 

1378 

1097 

5586 

2340 

#  Distinct 
Providers 

518 

246 

192 

122 

214 

132 

88 

158 

65 

#  Distinct 
Departments 

226 

80 

37 

44 

37 

35 

31 

37 

28 

#  Distinct 
Specialties 

33 

6 

3 

4 

3 

4 

3 

3 

3 

#  APRN 
Encounters 

3331 

25145 

2543 

159 

180 

282 

46 

245 

2410 

#  MD 

Encounters 

79997 

178942 

20527 

1316 

922 

2472 

1978 

9118 

3532 

#  DO 

Encounters 

6425 

16270 

2374 

158 

3442 

223 

261 

662 

40 
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Figure  1 .  Percentage  of  patient  encounters  for  obesity,  influenza  vaccination  and  gastroesophageal 
reflux  disease  at  obtained  values  of  A-t . 
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Figure  2.  Percentage  of  patient  encounters  for  encopresis/constipation,  headache  and  concussion  at 
obtained  values  of  A-t. 


17 


120 


- Supracondylar 

Fracture 

Idiopathic  Scoliosis 

. Type  1  Diabetes 

Miletus 


Figure  3.  Percentage  of  patient  encounters  for  supracondylar  fractures,  idiopathic  scoliosis  and  type  1 
diabetes  mellitus  at  obtained  values  of  A-t. 
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Figure  4.  Maximum  score  on  the  Concussion  clinical  knowledge  metric  as  a  function  of  values  of  the  A -t 
Statistic.  Model  F  (2,  134)  =  132.56;  p  <  .0001 
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ABSTRACT 

Military  and  civilian  healthcare  is  undergoing  radical  transformations  in  almost  every  aspect  of  patient  care  from 
diagnosis  to  treatment.  Along  with  increased  complexity  in  the  technology  of  delivery  systems  and  procedures, 
medical  knowledge  is  expanding  at  an  ever-increasing  rate,  and  yet  clinicians  are  expected  to  retain  knowledge  and 
remain  proficient  in  their  fields.  Frequency  of  exposure  to  specific  clinical  problems  and  processes  are  known 
contributors  to  physicians’  decay  of  clinical  knowledge  and  proficiencies.  For  example,  while  deployed,  military 
physicians  may  experience  less  demand  for  specific  clinical  skills  and  are,  therefore,  at  risk  for  knowledge  decay.  A 
systematically  applied  knowledge  retention  program  integrated  with  continuous  training  is  one  possible  response. 
However,  institutionalizing  standardized  training  at  fixed  intervals  for  all  may  not  be  the  most  cost-effective  nor 
efficient  solution.  This  paper  discusses  the  progress  of  a  research  study  tasked  to  develop  and  validate  efficient 
interventions  to  mitigate  physician  knowledge  decay  that  address  both  increased  domain  complexity  and  lower 
frequencies  of  exposure. 

The  process  of  intervention  selection  is  based  on  the  analysis  of  elements  of  the  care  for  nine  targeted  clinical 
problems  that  reveal  physician  knowledge  decay  with  decreasing  frequency  of  exposure  to  those  clinical  problems. 
Once  the  most  critical  elements  of  the  care  process  have  been  identified,  we  apply  a  structured  approach  for 
selecting,  developing,  and  evaluating  possible  interventions  geared  towards  choosing  those  that  specifically  address 
identified  knowledge  needs  and  align  with  the  organization’s  learning  goals,  infrastructure  and  operating  budgets. 
Recommendations  for  a  systematic,  yet  flexible,  method  for  evaluating,  weighing  and  scoring  multiple  knowledge 
decay  mitigation  alternatives  are  included,  supporting  interventions  ranging  from  static  job  aids  to  immersive 
learning  simulations.  In  summary,  this  paper  proposes  a  comprehensive  selection  model  for  continuing  medical 
education  programs  committed  to  prevent  skill  decay,  aid  knowledge  retention  and  improve  overall  physician  and 
organizational  performance. 
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INTRODUCTION 

Today,  the  healthcare  field  is  undergoing  radical  transformations  in  almost  every  aspect  of  patient  care.  Medical 
knowledge  is  increasing  so  rapidly  that  it  is  virtually  impossible  for  any  health  care  provider  to  master  it,  digest  new 
knowledge,  or  remember  all  that  has  been  learned.  Optimal  maintenance  and  retention  of  physicians’  clinical 
proficiency  could  reduce  costs  and  enhance  quality  of  care  (Halm,  Lee,  &  Chassin,  2002).  Regretfully,  physicians 
cannot  accurately  assess  their  own  knowledge  proficiency  (Davis,  Mazmanian,  Fordis,  Van  Harrison,  Thorpe,  & 
Perrier,  2006).  Within  this  environment  and  despite  the  best  intentions  of  individual  physicians,  degradation  of 
clinical  knowledge  and  skills  is  inevitable,  whether  it  is  compared  with  one’s  initial  knowledge  base,  the  knowledge 
and  skills  of  other  physicians,  or  performance  relative  to  published  guidelines. 

A  knowledge  retention  program  should  address  physicians’  knowledge  degradation  as  well  as  the  requirement  to 
continuously  reevaluate  and  update  their  training  and  education  programs.  Yet,  institutionalized  training  at  fixed 
intervals  may  not  be  the  most  efficient  nor  effective  solution  for  all  physicians  and  healthcare  organizations. 
Providing  solutions  based  on  when,  where,  and  how  to  develop  and  deliver  timely  interventions  is  becoming  as 
complex  as  the  processes  they  are  trying  to  improve. 

This  paper  presents  our  findings  in  developing  a  structured  approach  to  plan,  develop,  and  evaluate  physician 
education  programs  geared  towards  providing  personalized,  evidence -based  interventions  that  align  with  the 
organization’s  goals,  infrastructure,  and  operating  budgets.  It  is  an  approach  that  considers  different  factors,  such 
as:  level  of  intrusion,  feasibility,  compatibility,  interoperability,  maintainability,  and  other  related  parameters.  The 
intervention  choices  can  range  from  simple  cue  cards/posters  and  other  job  aids  to  more  sophisticated,  immersive 
learning  simulations.  Examples  of  the  types  of  interventions  and  knowledge  retention  programs  considered  include: 

Training  and  Instruction  Systems 

•  Human  led  instruction  -  such  as  lectures,  workshops,  video  streamed  sessions 

•  Static  computer  based  training  -  e.g.,  online  slideshow,  online  video 

•  Interactive  computer  based  training  -  such  as  avatar/voice  based,  virtual  instructor 

•  Simulation  training  -  e.g.,  medical  manikin,  actor  patients,  scenario  based,  live  demonstrations 

•  Individual  feedback  and  mentoring 
Job  Aids  and  Performance  Support  Systems 

•  Visual  aids  -  such  as  posters  in  exam  rooms,  checklists/flowcharts 

•  Computerized  clinical  decision  support  system  -  electronic  health  record  embedded  order  sets,  alerts,  input 
template  based  tools 

The  selection  of  intervention  may  drive  organizational  courses  of  action,  changes  in  healthcare  system 
operations/daily  business,  and  can  even  affect  the  way  electronic  medical  record  (EMR)  data  is  processed  in  order  to 
improve  physician  performance  and  patient  outcomes.  A  key  objective  of  our  program  of  research  is  to  develop 
efficient  interventions  to  mitigate  physician  knowledge  decay  that  address  both  increased  domain  complexity  and 
lower  frequencies  of  exposure.  This  paper  presents  preliminary  project  findings  and  assertions  of  research  work  that 
is  in  progress. 
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TRAINING  AND  INSTRUCTIONAL  SYSTEMS 

The  goal  of  a  training  or  instructional  course  is  to  facilitate  the  act  of  learning  to  commit  and  retain  information  in 
long-term  memory.  The  concepts  and  facts  stored  in  long-term  memory  about  a  topic  constitute  knowledge. 
Instructional  psychologists  call  this  type  of  content  declarative  knowledge  because  it  is  easy  to  recall  and  articulate 
(Clark  R.  C.,  2010).  In  the  following  sections,  we  discuss  the  current  state  of  notable  types  of  medical  training  and 
instructional  programs. 

Continuing  Medical  Education  (CME) 

As  noted  two  decades  ago,  didactic  educational  sessions  such  as  courses,  conferences,  lectures,  workshops, 
seminars,  and  symposia  have  little  direct  impact  on  competence,  performance,  and  patient  health  status  (Davis, 
Thomson,  Oxman,  &  Haynes,  1995).  Educational  meetings  can  be  made  more  effective  by  incorporating  mixed 
interactive  and  didactic  formats,  and  focusing  on  outcomes  that  are  likely  to  be  perceived  as  important  (Forsetlund, 
2009).  Smaller  interactive  workshops  can  result  in  moderately  large  changes  in  professional  practice  (O’Brien, 
Freemantle,  Oxman,  Wolf,  Davis,  &  Herrin,  2001).  Delivery  of  these  interventions  at  fixed  times  and  locations  may 
not  reach  the  clinician  who  would  most  benefit  from  a  refresher. 

Increasingly,  educational  content  is  being  delivered  online  to  promote  the  retention  and  dissemination  of  medical 
knowledge  during  “teachable  moments”  when  the  learning  is  most  immediately  relevant  to  the  physician. 
Advantages  of  this  method  include  standardizing  educational  content  that  can  be  easily  distributed  to  a 
geographically  diverse  population.  Educators  have  the  opportunity  to  regularly  update,  enhance  and  extend  existing 
curriculum  and  learners  can  control  the  time  and  place  of  learning  (Ruiz  &  Mintzer,  2006).  Measuring  and  defining 
effectiveness  of  these  educational  interventions  can  be  difficult,  although  knowledge  and  self-efficacy  appear  to  be 
improved  (Reed  &  Price  E.G.,  2005)  (Stark,  Graham-Kiefer,  Devine,  Dollahite,  &  Olson,  2011). 

Professional  Certification 

Since  2010,  the  US  medical  specialty  boards  have  incorporated  practice  assessment  and  improvement  as  part  of  their 
requirements  for  ongoing  professional  certification.  The  process  of  maintenance  of  certification  is  intended  to  assure 
the  public  that  physicians  adhere  to  standards  of  continuous  learning  and  assessment  throughout  their  professional 
careers.  There  is  growing  evidence  that  participation  in  maintenance  of  certification  programs  focusing  on 
promoting  practice  assessment  and  improvement  positively  impacts  physicians'  performance  and  their  patients' 
outcomes  (Gorzkowski,  Klein,  &  Harris,  2014)  (Vernacchio  &  Francis,  2014)  (Wittich  &  Reed,  2014). 

Medical  Simulation-based  Learning 

Simulation  is  commonly  used  throughout  healthcare  enterprises  to  enhance  patient  safety,  education,  and  quality 
improvement.  Simulation-based  medical  education  provides  an  interactive  hands-on  training  modality  that  bridges 
the  gap  between  classroom  learning  and  real  life  clinical  experience.  The  use  of  simulation  creates  a  safe, 
confidential  learning  environment  that  offers  the  participant  the  opportunity  for  deliberate  practice  followed  by 
facilitated  feedback  and  reflection  on  performance  (Motola,  Devine,  &  Chun,  2013).  This  supportive,  encouraging, 
non-punitive  environment  allows  learners  to  see  the  outcomes  of  their  mistakes  thus  gaining  powerful  insight  into 
the  consequences  of  their  actions  and  the  need  for  improved  performance  (Ericsson,  2004). 

At  Nemours  Healthcare  System  (NHS),  simulation-based  offerings  are  provided  for  attending  physicians,  fellows, 
residents,  nurses,  allied  health  care  providers,  non-clinical  associates,  patients  and  families.  The  goal  is  to  enhance 
participants’  technical,  cognitive  and  behavioral  skills  as  we  improve  quality  of  care,  systems,  processes,  and  spaces. 
In  2014,  6,588  participants  completed  645  educational  offerings  in  NHS.  The  medical  simulation  program  has  been 
used  in  training  and  ongoing  refresher  of  training  for  response  to  medical  emergencies.  Additionally,  an  outreach 
program  to  primary  care  practitioners  in  a  community  setting  engages  physicians  and  their  staff  in  responding  to  a 
child  with  an  acute  asthma  attack,  status  epilepticus,  and  other  clinical  scenarios. 
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Training  and  Instructional  Intervention  Choices 

Table  1  below  summarizes  the  key  elements  of  the  types  of  training  and  instructional  systems  that  our  research  team 
considered  in  support  of  physicians’  “maintenance”  of  knowledge  pertaining  to  management  of  targeted  clinical 
problems. 

Table  1.  Training  and  Instructional  Systems 


Type  of  Training  Media 

Description 

Examples 

Human-led  Training 

Training  delivered  in  the 
classroom,  laboratory  or 
video  recorded  media. 

Clinical  expertise  growth 
through  mentoring  -  protege 
relationship 

•  Group  sessions  /  workshops  /  lectures  in 
classroom  or  conference  room 

•  Thought  experiment  /  open  discussion 

•  Laboratory-based  lesson  or  tutoring 

•  Video-streamed  instruction  or  teleconference 
(live  or  pre-recorded) 

•  Feedback  -  by  Peer  or  Supervisor 

Basic  Computer-based 
Training 

Fixed  learning  content 
presented  through  a  computer 
program  or  browser 

•  Online  training  modules  or  workbooks 

•  Slideshows  or  series  of  framed  images  in 
sequence  to  simulate  animation 

Interactive  Computer- 
based  Training 

Computer  based  training  that 
can  present  different  content 
based  on  user  inputs 

•  Web-based  distributed  learning 

•  Interactive  courseware  and  testing 

•  Immersive  3D  simulation 

•  Avatar/virtual  instructor 

•  Virtual  or  augmented  reality 

Live  Simulation  Training 

Live  and  simulation  based 
medical  training 

Simulations  of  real  events  or 
imaginary  scenarios 

•  Actor-based  simulation  (actor  patients) 

•  Medical  manikin 

•  Scenario-based  live  exercises 

•  Life-size  immersive  3D  visualization 

•  Live  demonstrations 

JOB  AIDS  AND  PERFORMANCE  SUPPORT  SYSTEMS 

In  contrast  to  training  and  instruction  systems,  job  aids  and  performance  support  systems  are  designed  to  assist  a 
physician  while  executing  a  clinical  management  task.  When  applying  job  aids  and  performance  support  systems, 
organizations  must  provide  concise  information  for  the  physician  that  is  relevant  to  the  execution  of  real  time 
clinical  care. 

Job  Aids 

Russell  (1997)  makes  the  distinction  that  there  are  three  primary  components  of  a  job  aid: 

1.  A  job  aid  stores  information  or  instruction  external  to  the  user. 

2.  A  job  aid  guides  the  user  to  perform  the  task  correctly. 

3.  A  job  aid  is  used  during  the  actual  performance  of  the  task. 

A  job  aid  provides  a  visual  cue  to  provide  concise,  relevant  information  at  the  point  of  care  to  augment  physician 
memory  (Willmore,  2006).  It  is  important  to  point  out  that  job  aids  are  different  from  tools.  A  tool  bypasses  a 
physician’s  memory  involved  in  executing  a  task  or  may  complete  a  task  automatically.  That  is,  a  tool  does  not 
store  information  that  is  to  be  remembered  but  facilitates  performing  a  task.  Since  knowledge  retention  is  not 
enhanced  by  tools,  the  consideration  of  tools  as  an  intervention  is  not  part  of  our  study. 

Clinical  checklists  are  a  form  of  job  aids  that  have  been  adopted  from  the  successful  safety  interventions  developed 
to  prevent  pilot  errors  in  the  aviation  industry.  Simplified  clinical  checklists  have  been  popularized  by  Dr.  Atul 
Gawande,  a  surgeon  and  public  health  researcher  (Gawande,  2010).  Indeed,  implementations  by  the  World  Health 
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Organization  surgical  safety  and  safe  childbirth  checklist  programs  for  healthcare  providers  have  shown  to  improve 
clinical  outcomes  and  quality  of  healthcare  (WHO,  2014).  In  another  example,  a  simple  checklist  for  the  use  of 
contact  precautions  for  healthcare  providers  interacting  with  hospitalized  patients  resulted  in  a  40%  reduction  in  the 
incidence  of  healthcare-associated  Clostridium  difficile  infections  (Abbett,  et  al.,  2009). 

Electronic  Performance  Support  System  (EPSS) 

Computerized  Clinical  Decision  Support  Systems  (CCDSS)  are  a  form  of  EPPSS  that  provides  clinicians,  staff, 
patients  or  other  individuals  with  knowledge  and  person-specific  information,  intelligently  filtered  or  presented  at 
appropriate  times,  to  enhance  health  and  health  care.  CCDSS  encompasses  a  variety  of  tools  to  enhance  decision¬ 
making  in  the  clinical  workflow.  These  tools  include  computerized  alerts  and  reminders  to  care  providers  and 
patients;  clinical  guidelines;  condition- specific  order  sets;  focused  patient  data  reports  and  summaries; 
documentation  templates;  diagnostic  support,  and  contextually  relevant  reference  information,  among  other  tools 
(Clinical  Decision  Support,  2013).  CCDSS  can  improve  processes  of  care  including  diagnosis,  management  and  the 
quality  of  documentation.  CCDSS  provide  clinicians  with  computer-generated  clinical  knowledge  and  patient- 
related  information,  intelligently  filtered  or  presented  at  appropriate  times,  to  enhance  patient  care.  Clinical  decision 
support  can  help  practitioners  operationalize  treatment  guidelines.  Key  design  features  include  what  (information), 
who  (recipient),  how  (intervention  type),  where  (information  delivery  channel)  and  when  (in  the  workflow).  This 
framework  constitutes  the  Clinical  Decision  Support  Five  Rights  model  (Osheroff,  2009). 

In  practice,  adoption  of  a  CCDSS  can  be  challenging  and  can  have  unintended  adverse  effects  (Singh,  2010)  (Benin, 
2011)  (Eslami,  2006).  Researchers  at  Nemours  and  Yale  University  found  workflow  constraints,  technical  expertise, 
impediments  to  communication,  and  applicability  of  clinical  practice  guidelines  to  medically  complex  patients  were 
barriers  to  CCDSS  use  (Lomotan,  2012).  Physicians  commonly  ignore  CCDSS  generated  alerts  about  inappropriate 
medication  prescriptions  or  medication  combinations  that  could  cause  adverse  drug  reactions  and  CCDSS  do  not 
ensure  timely  follow-up  of  abnormal  laboratory  test  results  (Lin,  2008)  (Raebel,  2007).  In  the  United  Kingdom, 
these  types  of  lapses  have  been  addressed  through  the  combination  of  near  universal  use  of  electronic  health  records 
combined  with  CCDSS  and  substantial  pay  for  performance  incentives  (Zhou,  et  al.,  2009).  The  added  practice  of 
requiring  practitioners  to  justify  overriding  CCDSS  generated  advice  and  providing  the  advice  to  both  physician  and 
patient  appear  to  improve  CCDSS  effectiveness  (Roshanov,  et  al.,  2013)  (Seidling,  et  al.,  201 1). 

A  primary  goal  of  our  research  is  that  the  selection  of  the  intervention  or  intervention  combinations  is  tailored  to  the 
specific  body  of  knowledge  to  be  retained.  To  do  this,  we  need  to  consider  a  full  complement  of  intervention 
choices.  Table  2  summarizes  the  key  elements  of  job  aids  and  performance  support  systems  considered  for  the 
study. 

Table  2.  Job  Aids  and  Performance  Support  Systems 


Type  of  Training  Media 

Description 

Visual  Aids 

Posters  in  exam  rooms,  checklists/flowcharts,  etc. 

•  Posters  and  visual  aids  in  exam  rooms 

•  Posted  checklists 

•  Lanyards,  buttons,  ribbons,  badge  cards  with  acronyms  or  other  reminders 

Electronic  Performance 

Support  System  (EPSS) 

Computerized  Clinical  Decision  Support  Systems  (CCDSS) 

•  Check  lists 

•  Order  Sets  /  Smarts  Sets 

•  Best  practice  alerts 

•  Template-based  clinical  documentation  and  forms 

•  Emails  /  text  messages  reminders 

Online  Coaching  Material 

Sequenced  material  on  a  mobile  device  or  tablet  or  online  movie/animation 
clips  for  coaching 
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HUMAN  PERFORMANCE  CONSIDERATIONS 

A  careful  cognitive  and  human  performance  analysis  requires  evaluation  of  effective  methods  for  improving  human 
performance.  The  emphasis  on  improving  human  performance  requires  a  new  perspective  for  approaching  and 
solving  organizational  performance  requirements  beyond  limiting  the  solution  to  closing  the  knowledge  and  skills 
gaps  through  indiscriminant  instructional  solutions.  Rather,  evaluation  of  potential  solutions  should  include 
consideration  of  non-instructional  issues  that  affect  the  performance  of  individuals  and  teams  in  meeting 
organizational  goals  and  objectives.  These  non-instructional  factors  can  be  barriers  to  realizing  desired  human 
performance  outcomes  and  denigrate  the  effectiveness  of  an  instructional  solution  if  they  are  not  identified  and 
ameliorated.  Examples  of  non-instructional  factors  include:  the  clinical  setting,  number  of  patient  appointments  per 
day,  and  on-call  responsibility. 

Identification  of  Key  Knowledge 

For  the  study,  we  developed  a  standardized  process  that  enabled  us  to  evaluate  each  targeted  clinical  problem  (TCP) 
comparably.  The  process  applies  a  modified  cognitive  task  analysis  (CTA)  to  determine  the  specific  clinical 
knowledge  associated  with  each  problem.  This  involved  a  review  of  published  clinical  practice  and  Nemours 
Healthcare  System  organizational  guidelines  combined  with  interviews  with  clinical  specialists  and  experts.  The 
modified  CTA  focused  knowledge  elicitation  on  identifying  the  key  declarative  knowledge  associated  with 
diagnosing,  treating  and  reporting  the  clinical  condition  in  the  EMR. 

Knowledge  Metrics  from  Electronic  Medical  Records  Data 

Knowledge  identified  during  the  interview  and  subsequent  analyses  were  used  in  a  review  of  the  EMR  data 
“dictionary”  to  identify  fields  that  corresponded  to  the  required  knowledge  determined  by  the  CTA.  In  addition, 
queries  were  written  and  executed  to  locate  expressions  of  required  declarative  knowledge  in  EMR  free -form  text 
fields  such  as  progress  notes  and  patient  information. 

Knowledge  Utility  Function  and  Fit  Decay  Model 

Queries  extracting  knowledge  metrics  from  physician  free-form  text  logs  and  other  EMR  fields  were  combined  into 
a  multivariate  knowledge  utility  function  (KUF)  and  exercised  against  a  database  of  historical  patient  encounters  to 
establish  a  knowledge  decay  model  represented  by  Equation  1 : 

K(At)  =  e~yt  (1) 

where  t  is  time  since  previous  encounter,  K(Af)  is  the  KUF  value  where  y  is  the  vector  sum  of  weighted  individual 
differences  and  e  is  Euler’s  constant.  The  cognitive  sciences  recognize  this  as  an  application  of  the  Ebbinghaus 
knowledge  retention  model  (Ebbinghaus,  1999).  The  time  since  last  encounter  applied  in  the  equation  above,  with 
the  individual  gamma  value  produces  a  predicted  knowledge  level  K(t).  When  that  forecasted  level  drops  below  a 
set  point  at  an  associated  At,  we  trigger  a  need  for  a  targeted  intervention.  The  threshold  is  determined  through 
recommendations  from  subject  matter  experts  for  each  type  of  clinical  problem.  Note  that  this  approach  does  not 
differentiate  between  the  levels  of  experience  of  the  practicing  physicians. 


SELECTION  OF  INTERVENTIONS  BASED  ON  CONDITION 

Over  the  course  of  a  year,  a  multidisciplinary  team  consisting  of  physicians,  educators,  social  scientists,  health 
informaticists,  information  systems  experts,  researchers  and  administrators  met  in  person  thrice  with  six  interim 
teleconference  meetings  to  determine  a  structured  approach  for  planning,  developing,  and  evaluating  possible 
interventions  to  mitigate  knowledge  decay  and  promote  retention.  Key  determinants  of  intervention  selection  were 
found  to  be  the  primary  knowledge  elements  within  each  condition  to  be  addressed  and  decisions  around 
development  of  training  and  instruction  systems  versus  job  aids  and  performance  support  systems.  When 
developing  interventions  in  anticipation  of  knowledge  decay  or  at  the  point  of  care  to  mitigate  an  unmet  information 
need,  we  considered  several  factors  following  guidelines  suggested  by  Russell  (1997): 
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•  Frequency  of  knowledge  recall  -  Frequent  versus  infrequent  condition  encounters,  medical  seasonality  -  e.g. 
influenza  or  bronchiolitis  in  the  winter;  scoliosis  identification  with  annual  school  physical  examination 

•  Complexity  -  Multiple  or  few  decision  points,  extent  of  body  of  knowledge  -  e.g.  determining  appropriate 
management  of  gastroesophageal  reflux. 

•  Frequency  of  clinical  procedure  information  change  -  e.g.  influenza  vaccine  protocols  vary  from  year  to  year 

•  Consequence  of  error  and  error  proneness  -  Criticality  of  committing  and  likeliness  of  negative  results  -  e.g. 
determination  of  nerve  impingement  in  supracondylar  fracture  reduction 

•  Work  environment  and  variety  of  locations  related  to  point  of  care  -  e.g.  saturating  work  surfaces  and  walls  with 
brief  reminders  would  likely  be  ignored  or  may  not  be  feasible  in  all  locations 

As  we  apply  the  knowledge  utility  function  ( KUF ),  a  cause  analysis  is  performed  to  determine  the  type  of  knowledge 
decay  for  selection  of  intervention.  We  determine  if  the  lack  of  performance  is  due  to  a  deficiency  in  environmental 
support,  e.g.  data,  information  and  feedback  or  a  lack  of  repertory  of  skills  and  knowledge  and  individual  capacity. 
If  the  necessary  knowledge  can  be  presented  effectively  in  the  environment,  a  job  aid  is  recommended.  If  the 
exercise  of  knowledge  must  be  performed  in  situ  with  little  time  to  consult  a  job  aid  or  at  different  locations,  then 
training  is  indicated.  The  breadth  of  knowledge  required  in  the  medical  profession  must  also  be  considered.  A 
multitude  of  job  aids  can  quickly  become  a  hindrance  to  performance.  Our  recommended  intervention  selection 
flow  is  summarized  in  Figure  1 . 


Figure  1.  Intervention  Selection  Flow 
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The  research  team  developed  a  systematic  process  with  the  objective  of  ensuring  that  the  intervention  type  selected 
to  improve  care  for  each  of  the  targeted  clinical  problems  are  matched  to  empirically  identified  needs.  This  process 
employs  several  sequential  stages  for  each  targeted  clinical  problem  including  the  review  of  existing  EMR  data, 
stakeholder  surveys,  focus  groups  and  pilot  testing  explained  as  follows: 

1 .  Review  of  Existing  EMR  Data  for  Metrics 

The  team  reviewed  existing  EMR  data  to  specify  elements  of  the  optimal  care  process  for  each  targeted  clinical 
problem  (TCP)  in  search  of  metrics  that  tend  to  demonstrate  decay  in  association  with  increased  values  of  the  A-t 
statistic.  For  each  TCP,  the  team  fitted  a  multivariate  model  that  included  characteristics  of  the  worker  (e.g.,  title, 
department,  specialty,  proficiency  of  EMR  utilization),  characteristics  of  the  workplace  (e.g.,  clinical  setting, 
number  of  patient  appointments/day,  on-call  responsibility)  as  predictors  of  decline/maintenance  of  clinicians’ 
decision  making  proficiency  as  a  function  of  the  A-t  statistic.  This  multivariate  modeling  is  used  to  identify 
individual  health  care  providers  who  may  be  most  prone  to  clinical  skill  decay  and  to  guide  the  team’s  selection  of 
specific  EMR  data  elements  that  will  become  intervention  foci. 

2.  Establish  Criteria  using  Stakeholder  Surveys 

Stakeholders  (physicians,  health  informaticists,  administrators,  researchers,  educators  and  information  systems 
experts)  were  asked  to  complete  a  survey  to  quantify  their  perspectives  of  ten  dimensions  that  characterize  any 
intervention  (feasibility,  intrusiveness,  effectiveness,  cost,  development  time,  compatibility,  scalability, 
interoperability,  extensibility,  maintainability).  Among  40  stakeholders  there  was  a  clustering  of  responses  in  a  5 
choice  Likert  scale  which  were  further  collapsed  to  three  weights  (High,  Medium,  and  Low)  and  assigned  as  shown 
in  Table  3. 

Table  3.  Intervention  Decisional  Matrix  with  Stakeholder  Weights 


Domain 

Criteria  Questions  (Scored  1-5  points) 

Weight 

Feasibility 

1)  How  feasible  is  the  intervention  in  the  current  work  environment? 

High 

Intrusiveness 

2)  How  intrusive  would  the  intervention  be  in  the  current  work  flow? 

High 

Effectiveness 

3)  What  is  the  anticipated  effectiveness  of  the  intervention  in  mitigating 
knowledge  decay  based  on  comparative  efforts? 

High 

Cost 

4)  What  is  the  relative  cost  to  develop,  test  and  deploy? 

Medium 

Schedule  to 
Complete 

5)  What  is  the  estimated  time  to  establish  the  intervention? 

Medium 

Compatibility 

6)  Is  it  compatible  with  current  intervention  approaches  or  incompatible  / 
orthogonal  with  current  practice? 

Medium 

Interoperability 

7)  Can  the  intervention  be  integrated/connected  with  another  intervention? 

Medium 

Scalability 

8)  Can  the  approach  be  supported  enterprise-wide  or  limited  to  specific 
practitioners  or  clinical  locations?  Can  intervention  be  deployed  to  other  sites? 

Low 

Extensibility 

9)  Does  the  approach  consider  future  growth,  future  plans,  and  technology 
advancements?  How  tightly  coupled  and  limited  is  it  to  current  technology 
capability? 

Low 

Maintenance 

10)  What  is  the  estimated  effort  and  cost  to  maintain  and  sustain  over  the 
lifecycle? 

Low 

Table  3  shows  the  criteria  established  by  the  research  team  to  reach  a  final  recommendation  for  training  or  job  aids 
as  a  targeted  intervention.  Interventions  are  scored  on  a  scale  of  1  to  5  points  based  on  how  well  they  satisfy  the 
desired  answers  (e.g.,  a  highly  intrusive  intervention  receives  a  low  score  of  1,  versus  a  low  level  of  intrusion 
receives  a  5). 

It  is  important  to  note  that  these  criteria  should  be  applied  to  job  aid  selections  independently  from  training 
selections.  If  the  selection  is  made  combining  job  aid  and  training  choices,  then  a  job  aid  is  usually  the  preferred 
intervention  since  they  usually  can  be  developed  at  a  lower  cost,  in  shorter  development  time  frame,  with  little 
intrusiveness  and  a  quicker  worker  adoption  time.  Decision  makers  need  to  ensure  that  the  criteria  do  not  create  a 
bias  that  impedes  the  selection  of  training  interventions  when  they  are  necessary. 
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It  is  likely  the  priorities  of  stakeholders  in  aggregate  and  individually  may  vary  by  institution  which  would  impact 
on  the  weights  assigned.  It  was  remarkable  the  high  degree  of  alignment  at  this  one  health  system.  Proposed 
interventions  were  then  applied  to  this  decisional  matrix  and  stakeholders  rank-ordered  their  responses.  Note  that 
the  criterion  is  expanded  to  include  non-instructional  considerations  such  as  cost,  schedule,  scalability,  extensibility 
and  maintenance. 

3.  Establish  Focus  Groups 

Conducting  Focus  Group  meetings  with  stakeholders  (health  care  providers  representing  affected  clinical  divisions) 
helped  the  research  team  to  refine  their  perspectives  on  relative  weighting  of  elements  of  intervention  planning 
decisions  for  each  targeted  clinical  problem,  to  select  from  among  intervention  options  with  similarly  positive 
stakeholder  ratings,  and  to  combine  available  options  into  multi -component  interventions.  Acquiring  and  using 
detailed  stakeholder  perspectives  is  critical  to  ensuring  that  intervention  methods  are  selected  and  developed  by 
health  care  providers  for  health  care  providers  rather  than  being  imposed  upon  them  by  others  who  lack  a  detailed 
appreciation  for  the  complex  contexts  in  which  sophisticated  health  care  decisions  are  made. 

4.  Perform  Pilot  Testing 

Pilot  testing  of  intervention  methods  helps  the  research  team  to  obtain  preliminary  estimates  of  the  intervention’s 
feasibility,  acceptability  and  efficacy,  to  refine  the  intervention  accordingly  and  to  obtain  experience  with  it  before 
rolling  it  out  more  broadly.  Limited  experience  with  implementation  of  the  selected  interventions  before  a  larger- 
scale  rollout  will  enable  the  research  team  to  identify  and  resolve  potential  problems,  clarify  any  ambiguities  about 
implementation  and  confirm  the  capacity  to  evaluate  intervention  outcomes  empirically. 


EVALUATION  OF  INTERVENTIONS 
Assessment  of  Knowledge  Transfer 

In  evaluating  the  effectiveness  of  interventions  to  mitigate  knowledge  decay,  the  classic  evaluation  model  would 
propose  four  levels:  reaction,  learning,  behavior,  and  results  of  evaluation  (Kirkpatrick  D.  L.,  1998). 

•  Reaction  -  How  well  did  the  learners  like  the  learning  process? 

•  Learning  -  What  did  they  learn?  What  is  the  extent  to  which  the  learners  gain  knowledge  and  skills? 

•  Behavior  -  What  changes  in  job  performance  resulted  from  the  learning  process?  What  is  the  change  in 
capability  to  perform  the  newly  learned  skills  while  on  the  job? 

•  Results  -  What  are  the  tangible  results  of  the  learning  process  in  terms  of  reduced  cost,  improved  quality, 
increased  production  or  increased  efficiency? 

Post  intervention  learner  interviews  will  enable  evaluation  of  reaction  to  the  intervention.  We  plan  to  include  this 
throughout  the  intervention  development  phases.  Likewise,  training  interventions  will  include  an  immediate 
assessment  of  learning.  Evaluation  of  behavior  and  results ,  including  learning  transfer  will  be  obtained  using  post¬ 
intervention  EMR  data  entries  over  the  period  of  the  study. 

Measurements  from  EMR  Data 

Based  on  knowledge  retention  analytics  from  the  EMR,  we  propose  to  assess  the  following  performance  indicators: 

1)  Sustained  knowledge  retention  and  performance  proficiency  in  application  of  the  intervention  and  learned 
material  to  the  physician’s  immediate  job. 

2)  Consistency  in  the  expression  and  utilization  of  key  declarative  knowledge. 

3)  Generalization  and  adaptation  of  learning  to  jobs  or  tasks  not  originally  anticipated  by  the  training,  but 
related  in  a  way  that  allows  the  learning  effects  to  multiply. 

Measurement  of  individual  sustained  performance  utilizes  the  same  EMR  queries  and  metrics  used  to  develop  the 
learning  retention  model.  We  anticipate  a  return  to  standard  immediately  after  training  or  on-the-job  intervention. 
Consistency  in  knowledge  expression  should  also  stabilize  as  a  result  of  intervention  as  it  is  exercised  prior  to  a 
forecast  knowledge  decline.  We  expect  to  see  a  reduction  in  the  standard  deviation  of  the  aforementioned  metrics  in 
those  departments  where  the  intervention  is  deployed  as  compared  to  departments  without  the  intervention. 
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Metrics  based  on  EMR  data  for  each  of  the  selected  conditions  were  developed.  Since  we  propose  to  perform 
analysis  on  multiple  targeted  clinical  problems  (TCP),  we  may  be  able  to  observe  generalization  effects  of  improved 
expression  of  knowledge  with  the  other  clinical  conditions.  In  departments  where  an  intervention  is  applied,  effects 
may  be  observed  in  metrics  for  the  specific  condition,  as  well  as  for  other  conditions  due  to  a  crossover  effect. 


DISCUSSION 

This  paper  describes  preliminary  research  experiences  in  developing  and  implementing  a  systematic  strategy  for  the 
selection,  implementation  and  evaluation  of  interventions  designed  to  prevent,  minimize  or  mitigate  decay  in 
physicians’  clinical  declarative  knowledge.  We  propose  interventions  can  be  triggered  based  on  the  frequency  of 
exposure  to  the  TCP  of  interest  modified  by  clinician  and  workplace  environment.  The  EMR  will  serve  as  a  source 
for  detecting  risk  for  knowledge  decay  by  recording  TCP  exposure  and  presence  or  absence  of  key  clinical  data 
elements.  Downstream,  the  EMR  can  be  a  source  of  subsequent  effects  of  clinical  care  processes  and  outcomes. 

For  physician  assessments,  we  should  caution  about  solely  using  knowledge  as  expressed  in  the  EMR  as  a  measure 
of  clinical  performance.  The  application  of  declarative  knowledge  is  a  key  element  of  performance,  and  the  EMR  is 
limited  in  its  capability  to  indicate  the  presence  of  such  knowledge.  The  records  do  not  indicate  when  the 
knowledge  was  applied  by  the  physician  if  it  was  not  or  cannot  be  recorded  in  the  EMR.  Interpretations  of 
performance  and  intervention  effectiveness  should  be  understood  in  light  of  these  limitations. 

An  important  distinction  is  made  when  selecting  interventions  that  are  job  aids  versus  training  and  instructional 
programs.  Ideally,  most  instances  of  knowledge  decay  would  be  mitigated  with  job  aids,  since  these  tools  can 
supplement  clinical  knowledge  at  the  point  of  care.  Job  aids  in  the  form  of  check  lists  have  been  a  huge  success  in 
improving  performance  and  patient  outcomes.  However,  healthcare  organizations  need  to  consider  task  complexity, 
frequency  of  change  and  the  operating  environment  when  selecting  between  job  aids  and  training  interventions. 
These  factors  help  determine  whether  the  perceived  knowledge  decay  should  be  remediated  on  the  job  or  committed 
to  memory  via  training. 

Interventions  in  medical  training  and  operations  have  an  ultimate  goal  of  improving  the  quality  and  safety  of  health 
care  in  a  cost  effective  manner.  One  concept  to  be  explored  by  our  team  will  be  to  utilize  the  EMR  to  trigger  just  in 
time,  proactive  interventions  knowing  that  this  approach  is  limited  to  database  content.  In  the  future,  peak  physician 
performance  will  require  training  &  instructional  systems  and  job  aids  &  performance  support  systems  that 
dynamically  adapt  as  new  evidence  emerges,  provide  personalized  support  that  recognizes  the  clinician’s 
background  and  experience,  and  tailor  recommendations  based  on  specific  patient  characteristics.  Interventions  may 
also  be  customized  and  scaled  in  intensity  to  accommodate  a  wide  range  of  learners  from  novices  to  experience. 
The  expectation  is  that  just-in-time,  tailored  interventions  will  help  clinicians  not  only  gain  and  assess  critical 
knowledge,  but  also  increase  confidence  and  attain  enhanced  focus  when  encountering  more  complex  and 
demanding  aspects  of  patient  care. 
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Need  for  Cognitive  Performance  Models 
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Need  for  Cognitive  Performance  Models 

A  Management  Reguirement 


Successful  High  Reliability 
Organizations  (HRO) 
s  Maintain  quality 
■s  Control  production  costs 
■s  Meet  delivery  commitments 
■s  Learn  from  error  &  failures 

HRO  managers 

■s  Predict  &  controls  production  costs 

■s  Monitor  &  control  quality 

■s  Predicts  delivery  times 

■y  Consistently  delivers  useful  &  Success/ 

quality  products  Utility 

Success  increases  demand  for  HRO 


High  Reliability  Organization 

■■■fe 


Establishing  &  Keeping  the  HRO  effective  requires  performance  models 
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Need  for  Cognitive  Performance  Models 

A  Science  Problem  and  an  Art  -  Models  provide  the  Science 

•  Knowledge  is  dynamically  dependent  on  internal  &  external  processes 

-  Knowledge  is  not  a  constant 

-  Knowledge  varies  individually  across  time 

-  Time  is  a  proxy  for  unobserved,  processes  (e.g.  forgetting) 

-  It  varies  between  individuals  across  the  organization 

•  Knowledge  requirements  change  through  time  as  well 

-  Technology  requires  new  knowledge 

-  Science  &  discovery  produces  new  knowledge 

-  Standards  of  practice  change 

•  Planning  with  anticipation  of  knowledge  readiness 

-  Enables  decision  making  based  on  realistic  assumptions 

-  Helps  plan  affordable  training/retraining  schedule 

-  Helps  avoid  costly  errors  &  maintain  overall  quality 

•  Enabler  for  organizational  performance  monitoring 

-  Model  provides  assessment  metrics 

-  Model  identifies  realistic  performance  level  &  expectations 

•  Cognitive  performance  models  help  focus  maintenance  of  HRO 

-  identify  potential  areas  of  failure 

-  clarify  complex  processes  and  avoid  oversimplification 

-  emphasize  total  system  interdependency  and  operation 

-  provide  analysis  of  alternative  solutions  providing  resilience  under  adverse  situations 

-  identify  key  sources  of  critical  information  and  who  has  necessary  expertise 


Scientific  performance  models  enhance  critical  decision  making 
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Theory  of  Knowledge  Sustainment 


•  Background  Research 

•  Types  of  Knowledge  &  Performance 

•  Classes  of  Performance  Capabilities 

•  A  Learning  Curve  Model 

•  A  Knowledge  Decay  Model 

•  Knowledge  Decay  Factors 

•  Partitioning  the  Knowledge  Sustainment 
Model 
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Theory  of  Knowledge  Sustainment 

Background  Research 

•  Cognitive  Psychology 

-  Andersen ,  J.  R.,  Fincham,  J.  M.,  &  Douglass,  S.  (1999).  Practice  and  Retention:  A 
Unifying  Analysis.  Journal  of  Experimental  Psychology,  Learning,  Memory  and 
Cognition ,  1 1 20- 1136. 

-  Driskell,  J.  E.,  Willis,  R.  P.,  &  Cooper,  C.  (1992).  Effect  of  Overlearning  on  Retention. 
Journal  of  Applied  Psychology ,  615-622. 

-  Ebbinghaus,  H.  (1999).  Memory.  A  Contribution  to  Experimental  Psychology.  Bristol: 
Thoemmes  Press. 

•  Applied  Psychology 

-  Arthur,  Jr.,  W,  Day,  E.  A.,  Bennett,  Jr.,  W,  Portrey,  A.  M.  (2013).  Individual  and  Team  Skill 
Decay:  The  science  and  Implications  for  Practice,  New  York:  Routledge. 

•  Journal  of  Surgery 

-  Deering,  S.,  Rush,  R.,  Lesperance,  R.,  &  Roth,  B.  (2011).  Perceived  effects  of 
deployments  on  surgeon  and  physician  skills  in  the  US  Army  Medical  Department.  ” 
American  Journal  of  Surgery ,  666-672. 
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Theory  of  Knowledge  Sustainment 

Background  Research  (continued) 

•  Human  Performance  Technology 

-  Arthur,  Jr.,  W.,  Ben  net,  J.  W.,  Stanush,  P.  L.,  &  Me  Nelly,  T.  L.  (1998).  Factors  that 
influence  skill  decay  and  retention:  A  quantitative  review  and  analysis.  Human 
Performance ,  57-101. 

•  Skills  Deterioration  Symposium 

-  O’Neil  &  Rivera  (2011)  Educational  Psychology  and  Human  Factor  Issues  Involved  in 
Studying  (or  Evaluating)  Degradation  of  skills 

DOD  Research  (many  articles) 

•  Air  Force  Research  Laboratory 

-  Andrews,  D.  H.,  &  Fitzgerald,  P.  C.  (2010).  Accelerating  Learning  of  Competence  and 
Increasing  Long-Term  Learning  Retention.  Mesa,  AZ:  Air  Force  Research 
Laboratory/RHA,  Warfighter  Readiness  Research  Division 

•  Army  Research  Institute 

-  Wisher,  R.  A.,  Sabol,  M.  A.,  &  Ellis,  J.  A.  (1999).  Staying  Sharp:  Retention  of  Military 
Knowledge  and  Skills.  U.S.  Army  Research  Institute. 

-  Goodwin,  G.  A.  (2006).  The  Training,  Retention,  and  Assessment  of  Digital  Skills:  A 
Review  and  Integration  of  the  Literature.  Arlington,  VA:  US  Army  Research  Institute  for 
the  Behavioral  and  Social  Sciences. 
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Theory  of  Knowledge  Sustainment 

Types  of  Knowledge  &  Performance 


•  Declarative  Knowledge 

-  Defines  what  knowledge  is  needed  for  performance 

-  Is  Abstract,  often  described  as  book  knowledge 

•  Procedural  Knowledge 

-  Sequencing  of  actions 

-  When  is  order  important  and  not  important 

-  Performance  outcome  interdependencies 

•  Visual-Psychomotor  Abilities  (VPA) 

-  VPA  skills  acquired/retained  through  practice 

-  Learning  pedagogies  and  practice  builds: 

•  Accuracy  •  Speed  •  Quality 

•  Endurance  •  Strength  •  Fluency 


Study  Focus 


•  Attitudes  &  Affect 

-  Identifies  relative  importance  of  performance  effects 

-  Enables  response  agility  when  external  conditions  change 

•  Team/Leadership 

-  Team  Resource  Management 

-  Emotion  Management 


Skill  retention  is  complex  and  requires  multiple  brain  physiologies 
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Theory  of  Knowledge  Sustainment 

Classes  of  Performance  Capabilities 


-  Speed  -  Quality 

-  Strength  -  Fluency 


-  Memory  Aids 


Behavioral 

-  Accuracy 

-  Endurance 

Knowledge 

-  Memory 

Metacognitive 

-  Self  Monitoring 

-  Self  Respect  &  Confidence 

-  Honest  Self  Appraisal 

Cognitive 

-  Awareness  -  Focus  &  Flow 

-  Sensemaking  -  Decision  making 

Social 

-  Leadership  _  influence 

-  Coordination  _  Communication 

-  Language  &  Culture 

Adaptive 

-  Science,  Technology  &  Discovery 

-  Performance  Standard  Changes 


Knowledge  retention  assessment  must  be  multidimensional 
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Theory  of  Knowledge  Sustainment 

A  Learning  Curve  Model 


•  Subject  Matter 

-  Declarative  Knowledge  Requirements 

-  Procedural  Knowledge  Requirements 

-  Visual-Motor  Skill  Requirements 

-  Required  Level  of  Abstraction 

-  Decision  Making  Process  &  Support 

-  Team  /  Collaboration  Requirements 

•  Learner 

-  Intellectual  Development 

-  Emotional  Intelligence 

-  Meta-cognitive  Skills 

-  Prior  Experience 

-  Perception  of  Content  Relevancy 

•  Instructional 

-  Pedagogy 

-  Media 

-  Environment 


Learning  can  be  expensive  - 


ip  =  Maximum  knowledge  level 
p  =  Knowledge  metric  scaling  factor 
y  =  Knowledge  decay  shape  factor 
K(t)  =  Knowledge  at  time  (t)  =  qj  +  peyt 


Modeling  helps  optimize  that  investment 
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Theory  of  Knowledge  Sustainment 

A  Knowledge  Decay  Model 


Knowledge  decays  as  a 
function  of  time  between 
remembrances  (Ai) 

Conversely,  some  knowledge 
will  not  decay 

-  Knowledge  specific  decay 
may  be  a  function  of: 

•  Availability  Job  Aids 

•  Frequency  of  remembrance 

•  Similarity  to  unrelated  tasks 

•  Knowledge  acquisition  age  or 
technique 

-  Need  to  identify  knowledge 
that  does  not  decay 

Actual  knowledge 
performance  will  be  observed 
as  a  multivariate  Knowledge 
Utility  Function  (KUF) 


i  in0/. 

Knowledge  Decay 

1ZU/0 

100%  - 

80%  - 

> 

^  60%  - 

15 

to 

40%  - 

20%  - 

no/ 

\  K(At)  =  xp  +  pe-yAt 

\ 

A  ! 

At  — 

i 

i 

i 

i 

U/o 

- 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 - 1 — 

0  2  4  6  8  10  12  14  16  18  20  22  24  26  28  30  32  34  36  38  40  42  44  46  48  50  52 

Weeks w/out  Practice 

(f>=  Minimum  knowledge  level 

p  =  Knowledge  metric  scaling  factor 
y  =  Knowledge  decay  shape  factor 
K(Jt)  =  Knowledge  at  time  since  last  encounter 


KUF  obtained  from  observed  performance  data 
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Declarative  Knowledge  Modeling 
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Declarative  Knowledge  Modeling 

Model  Development  Requirements 


•  Select  Targeted  Clinical  Problem(TCP) 

-  Identify  relevant  clinical  conditions 

-  Select  a  subset  of  clinical  conditions  for  study 

-  Identify  knowledge  data  sources  for  these  problems 

•  Construct  Skill  Retention  Model  (SRM)  for  TCP 

-  Cognitive  Task  Analysis  (CTA)  to  identify  Knowledge,  Skills  &  Attitudes 
(KSA)  for  TCP 

-  Work  Performance  Metrics 

•  Mine  Electronic  Healthcare  Record  (EHR)  system  for  knowledge 
indicators 

•  Construct  performance  Metric(s)  for  TCPs 

•  Identify  confounding  concepts  &  develop  control  covariates 

-  CTA  EHR  utilization 

-  CTA  workplace  considerations 

-  Worker  Metrics 

•  Training  &  experience 

•  General  Abilities 

-  Workplace  Metrics  (external  considerations) 

•  Use  CTA  &  SRM  with  EHR  to  develop  retention  intervention  solution 


Skills  retention  is  a  function  of  the  work,  the  worker  and  the  workplace 
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Declarative  Knowledge  Modeling 

Model  Development  Process  -  Evidence-Based  Modeling 

•  Key  characteristics  a  linear  function  of: 

-  Task  characteristics 

-  Learner  capabilities 

-  Instructional  modes 

-  Performance  Environment 

-  Non-performance  environment 

•  Define  metrics  for  performance  analysis 

•  Create  Measurement  &  Assessment  Instrumentation 

•  Data  collection 

•  Model  Parameterization  Non-linear  Estimation  techniques 

-  SPSS,  SAS,  Jump,  R  &  MATLAB  statistical  packages  support  technique 
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Measuring  Knowledge  Retention 
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Measuring  Knowledge  Retention 


•  Knowledge  Utility  Function  (KUF) 
Development 

•  KUF  -  Institutional  Review  Board  (IRB) 
Compliance 

•  KUF  -  Data  Dynamic  Variation 

•  Model  Analysis  KUF  construction 
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Measuring  Knowledge  Retention 

Knowledge  Utility  Function  (KUF)  Development 


•  Determine  usable  data  from 
the  EHR 

•  Start  with  expert  identified 
knowledge 

•  Scan  EHR  dictionary  for 
relevant  data 

•  Evaluate  the  observable 

-  Is  it  IRB  compilable? 

-  Does  it  have  any  or  sufficient 
data? 

-  Does  it  vary  in  our 
population? 

-  Does  it  vary  according  to 
performance  time  gap? 
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Measuring  Knowledge  Retention 

KUF-  IRB  Compliance 


•  Metrics  derived  from  EHR  are  most  readily 
manipulated  into  IRB  compliance 

-  Identity  masking,  using  randomized  global  unique 
ID  (GUID) 

-  Event  time  masking  by  giving  time  as  offset  from 
undisclosed  start  time 

-  Data  grouping  (e.g.  transform  interval  data  into 
ordinal  quartiles) 

-  Excluding  data  from  study  that  are  not  compliable 

•  Individual  Metrics  based  on  other  sources 

-  Physician’s  Pulse 

•  EMR  skills  &  utilization 

-  Meaningful  Use 

-  Echo  credential  database 

•  Academic  credentials 

-  Data  grouping  w/  published  school  ranking 

-  School  location-State/non-US 

•  Country  of  Origin 

•  Language  skills 


June  2015 


Copyright  2015  Nemours  Children’s  Health  System 


21 


Measuring  Knowledge  Retention 

KUF-  IRB  Compliance 

Requires  IRB  approval  for  use... 

•  Metric  enables  identification  of  specific  patients 

-  Describes  identifiable  characteristic  of  specific  patients 

-  Observed  only  during  events  of  a  very  unique  nature 

-  Describes  existing  or  results  in  outcome  conditions  or  treatments  of 
unique  nature 

•  Metric  enables  identification  of  specific  physicians  or  other 
medical  practice  caregivers 

-  Uses  specifically  unique  skills  of  identifiable  physicians 

-  Observed  only  during  events  of  a  very  unique  nature 

-  Describes  existing  or  results  in  outcome  conditions  or  treatments  of 
unique  nature 

•  Metric  is  intrusive,  causing  direct  harm  to  physician  or  patient 

-  Even  metrics  derived  from  existing  EHR  or  other  sources  can  be 
intrusive  in  their  effects 
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Measuring  Knowledge  Retention 

KUF-  Data  Dynamic  Variation 


Not  usable  if... 

•  Data  has  no  variation  in  performance  frequency  (At=  0) 

-  Event  data  occurs  frequently  for  all  physicians 

-  Event  rarely  occurs 

•  Data  does  not  show  decay  over  time 

-  Performance  is  not  time  varying 

-  Performance  varies  greatly  at  all  times,  including  time  "i 0 ” 

•  Data  is  missing  for  some,  e.g.  At>  some  value 

-  Right  censoring  due  change  in  data  recording  process 

-  Left  censoring  due  to  limits  of  recording  process 

-  Systematic  missing  clustered  along  caregiver  attributes 
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Measuring  Knowledge  Retention 

Model  Analysis  KUF  construction 


•  Determine  if  performance 
data  can  be  made  continuous 

-  Use  Principal  Components 
Analysis  to  construct  latent 
variables 

•  If  small  number  of 
continuous  indicators 
normalize  and  use  them 

-  Determine  Max  value  of 
indicator(s) 

-  Fit  NLM  to  max  value 

•  If  indicators  dichotomous, 
transform  to  log  odds  and  fit 
NLM  to  value. 

•  Regress  on  individual  factors 

•  Apply  computed  weights  to 
forecast  model 
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Findings 
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Findings 

Observed  Obesity  Encounter  Rates  at  At 

•  Sample  taken  from  patient  encounters  during  2013 

•  Obese  defined  as  Body  Mass  Index  (BMI)  >  95th  percentile 

•  Across  Nemours  system,  90278  encounters  with  obese  patients 

-  Patient  seen  for  variety  of  reasons 

-  BMI  always  computed  — 

•  At  ranged  from  0  to  285  days 

•  57.76  %  occurred  on  same 
day  as  previous  obese  patient 
encounter 

•  Curve  is  a  gamma  fit  1 

-  implies  a  constant  arrival  rate 
of  obese  patients 

-  As  expected,  arrivals  occur 
frequently 

_  _  r  mm  0  4.5  9.0  13.5  18.0  22.5  27.0  31.5  36.0  40.5  45.0  49.5  54.0  58.5  63.0  67.5  72.0  76.5  81.0  85.5  90.0  94.5  99.0 

-  Note  presence  of  cases  of  At  I _ 2^ _ 

greater  than  two  weeks 


Variable:  Delta  t 
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Findings 

Knowledge  Utility  Function  (KUF)  computation 


•  Uses  17  factors  queried  from  EHR 

•  Factors  weighted  by  Eigenvalue  and 
summed  to  produce  Obesity  score 

•  Normalized  Obesity  Score 

-  Mean:  0.277753,  Standard  Dev:  0.17308, 
Min:  0.156525  Max:  1.45215 


Basic  Statistical  Measures 

Location 

Variability 

Mean 

0.277753 

Std  Deviation 

0.17308 

Median 

0.210812 

Variance 

0.02996 

Mode 

0.156525 

Range 

1 .29563 

Interquartile  Range 

0.17314 

Concept 

Variable  Name 

Mean 

Standard  Dev  Eigenvector 

Diagnoses  Obesity 

C Obesity DX 

0.329 

0.607 

0.4000 

Noted  Family  history  of  Obesity 

C_Obesity_FamHX_DX 

0.009 

0.113 

0.1004 

History  of  Obesity 

"c_Obesity_HX_DX 

ll002 

0.049 

0.0019 

Notations  on  abdominal  pain 

C Abdom Pain 

0.058 

0.276 

0.0079 

Notations  on  snoring 

C_Snoring 

0.095 

0.342 

0.1619 

Notations  on  apnea 

C_Apnea 

0.056 

0.266 

0.0947 

Notations  on  anxiety 

C Anxiety 

1X063 

0.293 

0.0338 

Notations  on  social  isolation 

C_Soc_lso 

0.002 

0.049 

0.0255 

Notations  on  activity  intolerance 

C_Act_lntol 

0.004 

0.074 

0.0760 

Notations  on  school  avoidance 

C School Avoid 

1X004 

0.071 

0.0174 

Notations  on  school  phobia 

C_School_Phobia 

0.005 

0.080 

0.0214 

Notations  on  shortness  of  breath 

C_Short_Breath 

0.004 

0.068 

0.0329 

Notations  on  sleepiness  during  day 

C_Day_Sleepy 

0.003 

0.064 

0.0413 

Advised  521  nearly  none 

C 521 

0.154 

0.496 

0.3740 

Noted  obseity  in  notes 

C_Obese 

0.261 

0.452 

0.5311 

Advised  Weight  Management 

C_Wt_Mngmnt 

0.080 

0.300 

0.4334 

Advised  Exercise 

C Exercise 

0.285 

0.571 

0.4167 

Variable:  ObesityScore 


0.15  0.21  0.27  0.33  0.39  0.45  0.51  0.57  0.63  0.69  0.75  0.81  0.87  0.93  0.99  1.05  1.11  1.17  1.23  1.29  1.35  1.41 

Obesity_Score 
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Findings 

Observed  Obesity  Diagnosis  Scores 


•  Individual  scores  did  not  vary 
with  At  as  expected  (e.g.  dashed  line) 

-  Substantial  variation  in  scores  at  all 
At ,  including  At-0 

-  Indicates  unobserved  processes  of 
adherence  variation 

-  Missing  values  =  525,  used  89753 
cases 

•  Plot  shows  predicted  95% 
confidence  limits 

-  with  n=89753,  p<.0001 


Estimation  Summaiy 

Method 

Gauss-Newton 

Iterations 

7 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Approx 
Pr  >  F 

Model 

1 

3.3223 

3.3223 

111.15 

<0001 

Error 

89751 

2682.7 

0.0299 

Corrected  Total 

89752 

2686.0 

Parameter 

Estimate 

Approx 
Std  Error 

Approximate  95%  Confidence 
Limits 

psi 

-2.2089 

0.000610 

-2.2101 

-2.2077 

rho 

2.4845 

qamma 

-0.00049 

0.000046 

-0.00058 

-0.00040 

K(At)  =  ip  +  pe  yAt 


Observed  Obesity_Score  by  Delta  t 

Obeshy_Score 


0  75  150  225  300 

At 
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Findings 

Observed  Max  Obesity  Diagnosis  Scores 


•  T ransformed  data  to  apply 
max  score  at  each  At 

-  N  =  105  -only  max  scores  included 

-  Applied  equal  weighting  to  each  At 

•  Regress  K(4f)  Max  Obesity 
diagnosis  scores  on  At 

-  Curve  shows  how  At  lowers  the 
score  of  everyone 

-  Plot  shows  predicted  95%  confidence 
limits  of  decay  curve 


Estimation  Summary 

Method 

Gauss-Newton 

Iterations 

14 

Source 

DF 

Sum  of  Squares 

Mean  Square 

F  Value 

Approx 
Pr  >  F 

Model 

2 

7.5384 

3.7692 

71.09 

<0001 

Error 

102 

5.4084 

0.0530 

Corrected  Total 

104 

12.9468 

Parameter 

Estimate 

Approx 
Std  Error 

Approximate  95%  Confidence 
Limits 

psi 

0.3375 

0.0454 

0.2475 

0.4275 

rho 

1 .0045 

0.0889 

0.8282 

1.1807 

gamma 

0.0372 

0.00717 

0.0229 

0.0514 

K(At)  =  ip  +  pe  yAt 


Observed  Maximum  Obesity  Scores  by  Delta  t 

Obesity  Score 

2.000  - 


-0.500- 

i  i  i  i  r 

0  75  150  225  300 

- m - 
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Findings 

Analysis  of  encounter  deviations  from  max  performance 

Evaluated  3  classes  of  factors  that  may  influence  individual  performance 

•  Work 

-  Time  with  patient:  appt_length  (in  minutes) 

•  Workplace 

-  Department  Specialty 

•  Dept_PCP  -  serves  Primary  Care  Pediatrics 

•  Dept_Weight  -  serves  Pediatric  Weight  Control 

•  Dept_Comorbid  -  serves  obesity  comorbid  conditions 

Cardiology,  nephrology,  endocrinology,  etc. 

-  Department  workload 

•  Dept_Vol  -  number  of  obesity  cases  in  2013 

•  ObesityLoad  -  ratio  of  obesity  cases  to  caregivers  in  department 

-  Department  Size  -  number  of  providers 

•  CareProviders 

-  Department  located  in  hospital:  Hosp_Setting 

•  Worker 

-  Physician  Specialty :  is  primary  care  or  not  -  PhyPC _ 


Variable 

Mean 

Std  Dev 

Minimum 

Maximum 

N 

Dept PCP 

0.3019113 

0.4590395 

0 

1 .0000000 

90255 

Dept Weight 

0.0466739 

0.2109513 

0 

1 .0000000 

90255 

Dept Comorbid 

0.2010526 

0.4007895 

0 

1 .0000000 

90255 

Dept Vol 

1505.54 

1284.95 

1 .0000000 

5067.00 

90255 

ObesityLoad 

217.2985373 

145.2268863 

1 .0000000 

598.0000000 

90255 

CareProviders 

7.9678799 

8.9905155 

1 .0000000 

60.0000000 

90255 

Hosp  Setting 

0.3502625 

0.4770547 

0 

1 .0000000 

90278 

APPT  LENGTH 

24.3088632 

17.4202364 

5.0000000 

430.0000000 

90273 

PhyPC 

0.3017014 

0.4589939 

0 

1 .0000000 

90273 
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Findings 

Analysis  of  encounter  deviations  from  max  performance 


Positive  deviation  =  better  performance 

Evaluated  factors  that  influenced 
individual  deviation  from  predicted  max 


Number  of  Observations  Read 

90278 

Number  of  Observations  Used 

89730 

Number  of  Observations  with  Missing  Values 

548 

performance 

-  N  =  89730 

-  P  <  .0001 

-  Adjusted  R-square:  0.1762 


Analysis  of  Variance 

Source 

DF 

Sum  of 
Squares 

Mean 

Square 

F  Value 

Pr  >  F 

Model 

9 

661.12823 

73.45869 

2133.33 

<.0001 

Error 

89720 

3039.40339 

0.03443 

Corrected  Total 

89729 

3750.53662 

Department  type  (Dept_PCP)  had  largest 

positive  effect :  +  1.01 

Physician  Specialty  in  Primary  Care 


Root  MSE 

0.18556 

R-Square 

0.1763  1 

Dependent  Mean 

-1.00817 

Adj  R-Sq 

0.1762 

Coeff  Var 

-18.40604 

(PhyPC)  had  largest  negative  effect:  - 0.76 


Parameter  Estimates 

Variable 

Label 

DF 

Parameter 

Estimate 

Standard 

Error 

t  Value 

Pr  >  |t| 

Standardized 

Estimate 

95%  Confidence  Limits 

Intercept 

Intercept 

1 

-1.15963 

0.00197 

-539.12 

<.0001 

0 

-1.16349 

-1.15577 

DeptPCP 

Dept PCP 

1 

0.45073 

0.05365 

3.40 

<.0001 

1.01203 

0.34558 

0.55538 

DeptWeight 

Dept Weight 

1 

0.13982 

0.00328 

42.65 

<0001 

0.14427 

0.13339 

0.14624 

DeptComorbid 

Dept Comorbid 

1 

0.07041 

0.00178 

39.49 

<0001 

0.13800 

0.06692 

0.07391 

Care  Providers 

CareProviders 

1 

0.00434 

0.00009713 

44.63 

<.0001 

0.18904 

0.00414 

0.00453 

ObesityLoad 

ObesityLoad 

1 

0.00011533 

0.00000313 

14.19 

<.0001 

0.08196 

0.00009944 

0.00013131 

Dept  Vol 

Dept Vol 

1 

-0.00002799 

8.88334E-7 

-31.51 

<.0001 

-0.17606 

-0.00002973 

-0.00002625 

PhyPC 

1 

-0.33633 

0.05369 

-6.27 

<0001 

-0.75613 

-0.44205 

-0.23160 

APPT LENGTH 

APPT LENGTH 

1 

0.00244 

0.00004076 

59.96 

<0001 

0.20810 

0.00236 

0.00252 

HospSetting 

1 

0.05284 

0.00146 

36.23 

<0001 

0.12325 

0.04998 

0.05570 
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Applications 
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Application 

Modeling  for  Performance  Optimization 

•  Gap  Analysis 

-  Cognitive/Social/Behavioral  Model  Analysis  identifies  critical  elements 

-  Assessment  tools  determine  readiness  state 

•  Personnel  Selection  &  Learning  System  Design 

-  Assessment  tools  for  applicant  screening  &  planned  learning  rate 

-  Optimize  media  &  pedagogy  for  minimum  time  to  readiness 

•  Used  to  Develop  /  Deploy  Performance  Support  /  Memory  Aids 

-  Cognitive  -  Memory 

-  Decision  Support 

-  Communication  aids 

•  Performance  Retention  Models  for  practice  schedule  design 

-  optimum  practice  frequency 

-  identifies  what  to  practice  &  when 

-  assessment  &  reinforcement  schedule  for  optimum  sustainment 

•  Models  assist  resource  assignment  to  optimize  performance 

-  Selection  of  “ready”  resources  for  critical  performance 

-  Guides  resource  rotation  to  maintain  readiness 


Performance  Modeling  Enables  Readiness 
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Modeling  for  Performance  Optimization 

Extending  to  system  performance  prediction  models 

•  Performance  models  are  scalable 

-  from  individual  to  teams  for  team  readiness  assessment 

-  from  teams  to  organizations  for  total  readiness  assessment 

•  Models  enable  distribution  of  resources  for  optimized  return  on 
investment 

•  Models  enable  distribution  of  reinforcement  schedules  at 
organizational  level 

•  Models  provide  metrics  for  managing  organizational  objectives 

-  Identify  possible  and  expected  performance  levels 

-  Assessments  identify  performance  shortfalls  where  application  of 
performance  improvement  actions  needed 

•  Models  provide  for  rational,  agile  management 

-  Enables  replacement  of  “we’ve  always  done  it  this  way”  with 
demonstrably  effective  alternatives 

-  Enable  maximization  of  affordability  goals 


Performance  Modeling  Enables  Affordable  Total  Force  Readiness 
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Skill  Retention  Issues  in  Education 

Classroom  to  school  system 

•  Procedural  performance  can  be  varied 
resulting  in  skill  retention  issues 

-  Teachers  doing  administrative  service 

-  Specialists  teaching  general  courses 

•  State  of  the  Art  and  Practice  is  in 
continuous  change 

-  New  media  and  instructional  techniques 

-  New  technologies  and  learning  content 

•  Data  may  exist  &  be  obtainable  to  properly  parameterize  models 

•  Certification  &  skill  maintenance  is  ongoing  &  recognized  issue 
within  the  industry 

•  Educational  performance  assessment  is  a  strong  political  issue 
in  local  settings 


r  ^ 

Performance  modeling  leads  the  way  to  enhancing  the  education  system 
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Healthcare 

Physicians ,  technicians  &  nursing 


•  Procedure  performance  can  be 
varied  resulting  in  skill  retention 
issues 

-  Specialists  doing  generalist  work 

-  Generalists  doing  specialist  work 

-  Randomness  &  frequency  of  illnesses 
&  injuries  define  experience  base 

•  EHR  data  often  exists  to  parameterize  models 

•  Certification  &  skill  maintenance  is  ongoing  recognized  issue 
within  industry 

•  Healthcare  Industry  seeks  models  for  quality  assurance 

-  Providers  and  practitioners 

-  Insurance  providers 

•  Serving  Military  physicians  and  reservists  have  recognized 
risk  of  skill  decay 
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Department  of  Defense 

Performance  modeling  across  the  Services 

•  Procedural  performance  &  deployments  can  be 
varied  resulting  in  skill  retention  issues 

-  Cultural  Engagement  is  complex  &  varies  from 
location  to  location 

-  Combat  skills  intermingled  with  leadership  & 
cultural  needs 

-  Need  for  diverse  skills  due  to  force  reductions 

•  State  of  the  Art  and  Practice  is  in  continuous  change 

-  New  technologies  &  agile  tactics 

-  Agile  enemy  requires  agile  response 

•  Much  data  may  already  exist  to  properly  parameterize  models 

•  Certification  &  skill  maintenance  is  ongoing  &  recognized  issue 
within  Services 

•  Performance  assessment,  AAR  technologies  &  performance 
modeling  are  recognized  requirements 


Evidence-based  performance  modeling  meets  a  growing  DOD  need 
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Our  customer  wants  optimum  performance 


Combined 
Performance 
Maintenance  Mod< 


*e 

Optimum 
T  Performance 


-  100% 


Z  50% 


Human 

Centric 

Design 


Learning  Curve 


Deliberate 
Methods  & 
Tools 


-  100% 


Z  50% 


K(At)  =  ip  +  peyAt 


Improvement 


Relearning  Curve 
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Performance  forecast  modeling  is  a  key  affordable  discriminator 
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Contact  Information 


Crutchfield  Jr,  James  H. 
iames.h.crutchfield.ir@lmco.com 

Ingraham,  Lorie  R 
lorie.r.inqraham@lmco.com 


Maria  Carmen  G  Diaz,  MD 
MariaCarmen.Diaz@nemours.org 

James  P  Franciosi,  MD 
James.Franciosi@nemours.org 

Jim  Miller 

James.Miller@nemours.org 

Lloyd  N  Werk,  MD,  MPH 
Llovd.Werk@nemours.org 

Tim  Wysocki,  Ph.  D.,  A.B.P.P. 
Timothv.Wvsocki@nemours.org 


Lockheed  Martin  Corporation 
Mission  Systems  and  Training 
100  Global  Innovation  Circle 
MP823 

Orlando,  FL  32825 


Nemours  Children’s  Health  System 
807  Children’s  Way 
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